?

Log in

No account? Create an account

Previous Entry | Next Entry

Sometimes your protection...

is the thing that lets you down

Just after eleven this morning my pager started buzzing with network management system messages telling me that central library had just dropped off the face of the Earth. I mean *VOOM*, everything we track there went down all at once. Generally, this is not a good indication.

I started making calls -- to Irene the library IT manager, and to the city architect in charge of the renovation project under way there -- and got nothing but voicemail. Two of my troops (Pat and Shawn) were en route to the Library when the helpdesk called me with word from Irene (and her personal cell number) ... I got her on the phone and got the news: the refrigerator sized UPS unit in their computer room had caught on fire. The fire department had killed power and was working to get the space vented out. While we were on the phone HFD gave the Library folks the OK to reenter the building.

Pat and Shawn arrived shortly thereafter, along with one of the City's electrical inspectors. He got the room powered up by bypassing the UPS. Everything came up alright, but without any power protection. We had a spare portable (135 lb, 2200VA) UPS in one of the satellite closets which Pat and Shawn lugged into the computer room. After lunch I helped them load another four 2200VA units into Shawn's truck to take over to the library. All of the library servers are now back in operation protected by our loaner UPSs.

The apparent cause of the burn-up is a generator test gone awry. Oddly enough, all of the construction staff who were involved in the generator test fled the scene when the Fire Dept rolled up. This will probably take some sorting out, especially as the electrical contractors who are working in the building now (phase II - renovating the old library building) are not the same ones who installed the generator and the UPS (part of the addition that was phase I). The fried UPS is probably in the $12,000 - $20,000 range, and you can be sure that none of the contractors will pay that amount of money willingly.

###

We had our own UPS story this morning, but it was far less serious. Mark-my-unindicted-co-conspirator and I were helping Kevin (one of my network engineers) install a rack-mount UPS over in a tiny little network closet in the basement of the Board of Ed building. Kevin had reversed the mounting brackets on the UPS so that instead of hanging in the rack with the front of the UPS flush with the face of the rack and all of the weight of the unit sticking way out the back, it would be centered front-to-back, keeping the weight in balance. This looked like a grand idea -- we got it mounted in the rack and all of the gear plugged in. And then Kevin tried to close the door. *THUNK* Did I mention that the door to this closet opened inward?

We pulled it out of the rack, repositioned the mounting brackets so that the back/front balance was more like 60/40 and screwed it back in. This time the door made its swing past the front of the UPS with an inch to spare!

Comments

( 2 comments — Leave a comment )
half_elf_lost
Jul. 1st, 2006 03:51 am (UTC)
Oddly enough, all of the construction staff who were involved in the generator test fled the scene when the Fire Dept rolled up.

That's because their supervisor told them all to disappear because some of them probably aren't licensed to do what they're doing. Let's hope someone asks for an itemized list of ALL contracted workers who were there.

In a similar area, the Advocate had an article last week wondering why that damn project still isn't finished.
netcurmudgeon
Jul. 1st, 2006 01:07 pm (UTC)
In a similar area, the Advocate had an article last week wondering why that damn project still isn't finished.

Well, the second phase is going much better than the first phase. The phase 1 addition sat unoccupied for over a year as the City battled the flooring contractor over the cork floor, whose tiles started popping loose within a month of installation.

That weak defense of the project aside, there is something terribly broken about how Hartford buildings things. There isn't one fault, IMO, but many.
  • Bid docs that are not written tightly enough allow unqualified contractors to win jobs.
  • Public Works is woefully understaffed, having lost 90% of their professional staff since 2000.
  • Corporation Council seems to be more interested in settling cases instead of holding contractors to what their bid specs require.
  • The minority/small business set-aside program brings in contractors that are too small and too inexperienced to do the jobs timely and properly.
  • There appears to be no senior executive oversight of these project in between the high-level political governance (e.g. the Hartford School Building Committee) and the professional staff charged with executing the work.

The net result is projects that drag on for years and are plagued by problems. The darkest sin of them all isn't the Library -- it's Hartford Public High School.
( 2 comments — Leave a comment )

Latest Month

January 2017
S M T W T F S
1234567
891011121314
15161718192021
22232425262728
293031    

Tags

Powered by LiveJournal.com
Designed by Lilia Ahner