News broke this morning that Glasgow City Council’s email is down because a safety measure they installed went rogue. An air conditioner in the server room malfunctioned, releasing gas. This in itself was an inconvenience but all IT systems continued to operate until the gas reached the fire suppression system.
The fire suppression system was unable to differentiate between air con gas and smoke, so started hosing down the servers. On Tuesday afternoon the Council took to Twitter to explain that their email system had been down since Monday. A day later there’s still no news of remediation. The failure also brought the telephone switchboard down for a period on Monday.
Air con systems are absolutely necessary for server rooms but we’ve seen similar instances to this too often. Systems designed to keep your servers safe, involving water, electricity and (often) fans running on an external wall, are notoriously prone to what Glasgow City Council yesterday called a ‘catastrophic’ failure.
The Council’s apparent lack of Business Continuity plans are another story. Getting this right isn’t difficult, or particularly expensive. Continuous image backup to a failover server elsewhere on the Council estate would have systems back in minutes. They could also outsource this risk to to Office 365.
