The Accounting system has caught up and is responding correctly. We consider the system recovered. As always, let us know if you see issues at support@opensciencegrid.org
Posted Sep 21, 2021 - 16:05 UTC
Monitoring
OSG Accounting service has recovered enough to start the web frontend. It is still scanning through replicas so the interface may be slower than normal, but it should respond with full results.
No data was lost due to the outage.
Posted Sep 21, 2021 - 14:41 UTC
Update
The accounting service has been started. It is now scanning the data before being fully available. It may take an hour or 2 to become generally available.
Posted Sep 21, 2021 - 13:57 UTC
Identified
A power outage has affected the data center hosting the OSG Accounting Services. The generator that is suppose to cover a power outage did not take over when the power was out, so the services only ran for the 15 minutes that the in-room batteries where able to power. The power was restored, but we now begin the long process of bringing back up servers and services. We expect that OSG accounting will be out overnight, until at least tomorrow (9/21).
Posted Sep 20, 2021 - 23:36 UTC
This incident affected: Accounting (GRACC Frontend, GRACC Backend, GRACC APEL Reporting) and Open Science Data Federation (Data Federation Accounting Service).