176 Minutes Without Power, A Datacentre Nightmare Part 2 ▶
On the 25th November 2021, the Walton Institute, South East Technological University (SETU) suffered an unplanned and total power loss in their Datacenter at the Netlabs Building, SETU West Campus Carriganore. This talk will cover what was observed before and during the outage, the downtime and damaged it caused, and the immediate aftermath. It will also detail the actions that were taken to keep everybody safe, to minimise the damage to hardware and services, the process of temporarily maintaining service continuity under adverse conditions. It will also cover the planning for and the installation of replacement infrastructure, which again, necessitated disconnecting the power supply from the building, removal of old UPS equipment and installation of new equipment while maintaining services for Staff, Students and external clients.
The Research Infrastructure and Testbed Services team of the Walton Institute are still picking up the pieces and some of the recovery processes are still ongoing. We would like to share some lessons learned, some mistakes to be avoided and interesting pictures of the event that even we had not seen before.