Interesting read. I worked for State Government here in Australia for 9 years, serving in the IT department at one of Brisbane's largest hospitals (RBWH at Herston) We too had a similar incident, although not as bad as yours. Like you, we were running blade servers, but hundreds of them. Infact maybe over a thousand at the time (I never counted them all) but our data centre was the size of about 3 full size basketball courts. Our data centre was fitted with a particle detector - these things were super sensitive. I mean, if you walked near one, and say brushed your arm, the digital read out on the front would change. Amazing technology.
Anyway, every now and then, the particle detectors would go off and sound the alarms. Being in a hospital this meant an entire evacuation of the building. Wed be heading downstairs and the fire dept would be on their way up. This went on for a few weeks until we detected a faulty power board on one of the server racks. (Funny enough you could kind of smell it) but being in such a huge data centre, it was almost impossible to find.
Anyway, in the end we managed to find it and replace it, but the entire operation, took months, just because of one small electrical smell/burnout from a powerboard. Being a government department, there were VERY strict about OHS and put about 1200 people through a Fire Safety and Training course with this company Fire & Safety Aus. It must have cost them an absolute fortune, but surely worth it. I think after several weeks of everyone running up and down 13 flights of stairs we were well and truly over it!
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
zvi
(no subject)
Date: 2015-01-16 11:18 am (UTC)Anyway, every now and then, the particle detectors would go off and sound the alarms. Being in a hospital this meant an entire evacuation of the building. Wed be heading downstairs and the fire dept would be on their way up. This went on for a few weeks until we detected a faulty power board on one of the server racks. (Funny enough you could kind of smell it) but being in such a huge data centre, it was almost impossible to find.
Anyway, in the end we managed to find it and replace it, but the entire operation, took months, just because of one small electrical smell/burnout from a powerboard. Being a government department, there were VERY strict about OHS and put about 1200 people through a Fire Safety and Training course with this company Fire & Safety Aus. It must have cost them an absolute fortune, but surely worth it. I think after several weeks of everyone running up and down 13 flights of stairs we were well and truly over it!