UW-Madison Machine Room Cooling Loss
Incident Report for Open Science Grid
Resolved
A sufficient number of hosts have been rebooted to restore services; marking incident as closed. We will continue to monitor over the weekend to check for services that are unstable.
Posted Nov 27, 2021 - 19:07 UTC
Investigating
The datacenter at UW-Madison hosting several OSG services lost cooling capacity overnight (starting at approximately 12:30am), resulting in several hosts going offline at 2:00am.

Services are currently being restarted.
Posted Nov 27, 2021 - 15:46 UTC
This incident affected: Software Repositories (Yum Repos), Hosted GlideinWMS (IGWN GWMS Frontend, JLAB GWMS Frontend, GLUEX GWMS Frontend, UCSD CMS GWMS Frontend, UCSD CMS VO Collector), Websites (Display), and Hosted CEs (Hosted CE Infrastructure).