netcurmudgeon (netcurmudgeon) wrote,

  • Mood:
  • Music:

Wet electrons are no fun

Some days it really doesn't pay to look at your pager. Walking into the office this morning I checked the messages that had accumulated overnight. Just after 02:30 Webster elementary school had dropped off the network all at once, and not come back. That's not the pattern for a power outage, and a look at CL&P's outage report page confirmed that there were no reported outages in Hartford. I intercepted Pat on the road and asked him to head straight for Webster.

A few minutes later he called me to say that he'd just heard from a supervisor at Buildings and Grounds: Webster had had a flood overnight. Now flood is not a good word in any context, but it's a very bad word in the context of electronic equipment.

Around eight thirty Pat called in to report that the flood had originated in the bathroom one floor up, right above the MDF. Our racks had taken a direct hit from the water that had been cascading down through the ceiling.

As Pat headed back to the office, Shawn and I hastily assembled spare parts -- including my one spare core switch. Pat, Shawn, and Sean from the phone group loaded up the department van and headed out. I followed a bit later after talking with the Acting Boss (who was in Boston at an E-Rate training seminar), the Superintendent's office, and the office of the Asst. Superintendent for Operations.

When I got on site I learned that the flooding had taken down one of the elevator machine rooms (right next to our MDF) and traveled to several other basement rooms. I later learned that water had also flooded the air ducts in the area. The cause was a hose that one of the custodians had left turned on and connected to a soap mixer; in the middle of the night the hose had burst away from the soap mixer and started gushing water at 85 PSI all over the 1st floor staff bathroom / custodial closet. The water overwhelmed the floor drain, and gravity started doing its thing.

We got primary phone service restored by 10:30 (half an hour ahead of my estimate, and 90 minutes ahead of my worst-case estimate -- I have great people). Total restoration took until around one. There is still repair work to be done. The voice patch panels that got soaked need to be ripped out and redone. Because of the craptastic installation job done by the low-bidder contractor, there are no service loops on any of the voice cables coming from the classrooms, so all of them will have to be torn out and re-pulled from scratch.

All told the deluge wrecked over $74,000 in equipment, and will cost an additional $10,000 in cable repair. When you add in possible damage to the elevator controller and the cleanup of the ducts, we could be talking a serious pile of money. The kicker is that this happened in the still new one year-old addition to the school. Because someone didn't turn off a faucet last night. I hope they dock that guy's pay from now until doomsday.

  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.