Just after eleven this morning my pager started buzzing with network management system messages telling me that central library had just dropped off the face of the Earth. I mean *VOOM*, everything we track there went down all at once. Generally, this is not a good indication.
I started making calls -- to Irene the library IT manager, and to the city architect in charge of the renovation project under way there -- and got nothing but voicemail. Two of my troops (Pat and Shawn) were en route to the Library when the helpdesk called me with word from Irene (and her personal cell number) ... I got her on the phone and got the news: the refrigerator sized UPS unit in their computer room had caught on fire. The fire department had killed power and was working to get the space vented out. While we were on the phone HFD gave the Library folks the OK to reenter the building.
Pat and Shawn arrived shortly thereafter, along with one of the City's electrical inspectors. He got the room powered up by bypassing the UPS. Everything came up alright, but without any power protection. We had a spare portable (135 lb, 2200VA) UPS in one of the satellite closets which Pat and Shawn lugged into the computer room. After lunch I helped them load another four 2200VA units into Shawn's truck to take over to the library. All of the library servers are now back in operation protected by our loaner UPSs.
The apparent cause of the burn-up is a generator test gone awry. Oddly enough, all of the construction staff who were involved in the generator test fled the scene when the Fire Dept rolled up. This will probably take some sorting out, especially as the electrical contractors who are working in the building now (phase II - renovating the old library building) are not the same ones who installed the generator and the UPS (part of the addition that was phase I). The fried UPS is probably in the $12,000 - $20,000 range, and you can be sure that none of the contractors will pay that amount of money willingly.
We had our own UPS story this morning, but it was far less serious. Mark-my-unindicted-co-conspirator and I were helping Kevin (one of my network engineers) install a rack-mount UPS over in a tiny little network closet in the basement of the Board of Ed building. Kevin had reversed the mounting brackets on the UPS so that instead of hanging in the rack with the front of the UPS flush with the face of the rack and all of the weight of the unit sticking way out the back, it would be centered front-to-back, keeping the weight in balance. This looked like a grand idea -- we got it mounted in the rack and all of the gear plugged in. And then Kevin tried to close the door. *THUNK* Did I mention that the door to this closet opened inward?
We pulled it out of the rack, repositioned the mounting brackets so that the back/front balance was more like 60/40 and screwed it back in. This time the door made its swing past the front of the UPS with an inch to spare!