Stash HTTP Access reporting 403: Permission Denied
Incident Report for OSG Consortium
Postmortem

The server allowing HTTP and StashCache access to the Stash public directories was reporting 403: Permission Denied errors. The node had previously filled it’s local storage space, leaving the Stash CEPH mount in an unusual state. The filled storage had been cleaned before this incident.

Rebooting the machine solved the bad CEPH mount and allowed normal downloads.

Posted Jan 11, 2021 - 17:19 UTC
Resolved
The stash.osgconnect.net system was restarted and downloads are now working.
Posted Jan 11, 2021 - 17:16 UTC
Identified
The issue has been identified as a bad mount on the stash.osgconnect.net host. Rebooting the machine in order to fix the mount.
Posted Jan 11, 2021 - 17:12 UTC
Investigating
We are currently investigating the issue with HTTP access to StashCache. This does not affect the posix mount on the login nodes, nor the StashCache federation access.
Posted Jan 11, 2021 - 17:11 UTC
← Current Status Powered by Atlassian Statuspage