After early morning fire-putting-out, I had a good rest of the morning. Pat and I were out and about hauling equipment out of a pair of schools back to our office. Naylor Elementary had become a UPS graveyard; we went out with a new unit and retrieved three dead UPSs along with an idle server. The server has got to be the most filthy server I have ever seen, but it had two 18GB drives in it! The main wiring closet at Naylor is in the gym teacher's office. This accounts for all the dirt and dust that the equipment is choked with. It also laid the groundwork for Pat and me to get a good eyebrow-rise out of the teacher.
Pat and I got there around 9:15, and PE was in full swing. We trundled right into the gym office and closed the door. Pat set to work unboxing the new UPS while I went behind the racks to pull out the three dead ones. The dust was so freaking thick that both of us donned dust masks. Ten minutes into our work the teacher came into to find some paperwork. We both look up and call out "good morning!" He takes one look at us and says "I'll get out of your way!" *Zoom*, back out into the gym with the door banging shut behind him.
After depositing our Naylor treasures back at the shop we headed out for Hartford Public High School to retrieve all the gear we took out of the wiring closet we demolished Thanksgiving week (HPHS is being added onto and renovated; work has progressed to the point where one of our wiring closets is getting wiped out). We took out twelve 10/100 Ethernet switches, about 100lbs of patch cables and three equipment racks. The switches and patch cables will make much needed additions to our skimpy stock of parts. We crated the switches and cables in a bunch of nursery crates I had brought in and stowed them in the bed of my truck. The racks we laid across the length of the bed and strapped down. I swear we looked like the IT Sanford and Son, only my truck didn't have enough dents!
The server problems from the wee hours of this morning were still tugging at the back of my brain when I got home. That's when I realized that my UNIX boxes in that computer room had been affected too. They'd bounced up and down several times during the same 01:30 - 04:00 time period. That conclusively ruled out a Windows server admin gone astray. What it indicated wasn't much fun though.
This particular computer room has two power feeds. One is regular commercial power from the building. The other is a feed from AT&T's facility three floors below. They have a generator, which we do not. Back in 2002 when the building was experiencing a rash of power problems due to ongoing construction work, my predecessor struck a gentlemans' agreement with the AT&T facility manager. AT&T provides us the equivalent of a 100 Amp feed. We split it between our critical communications equipment and critical servers in the computer room.
Digging through event logs and NMS records showed that the servers that went down were all on the AT&T feed. The servers on commercial power stayed up, but had errors in their event logs when the computer room ATM switch (also on AT&T power) went down. Strangely enough, the comm. equipment in the main wiring closet didn't go down. Either the draw on the UPSs there was low enough that they stayed up through the power disruptions, or the problem was some funky thing having to do with the three-phase power that comes up from AT&T.
Either way, I'm going to have an interesting time talking to AT&T tomorrow. You see, I have an inkling that since 2002 AT&T has converted their facility to unmanned operation. I don't even know if GB, the old facility manager, is still there (he was close to retirement in '02, and AT&T has had a bunch of layoffs in the past two years). Also, when my predecessor negotiated this arrangement, GB was acting on his own prerogative without consulting AT&T corporate. So, I could wind up having a conversation with someone from AT&T tomorrow where they say "we're supplying you with what?!?!"
It's never a dull day in Hartford.