netcurmudgeon (netcurmudgeon) wrote,

  • Mood:
  • Music:

First day back from vacation...

So, my first day back at work from vacation was ... a bit of a zoo.

The morning was spent on paperwork, finances, finding out who exactly got laid off last week, and more paperwork.

The afternoon was spent dealing with the worst network failure we've had since January of '06.

Our brand new Nortel ERS8306 routing switch -- the core of our data center, in service for only two weeks, but after a month of burn-in and bench time -- failed catastrophically at 13:24.

The new switch -- brought back from the dead with a loaner CPU card from Nortel

Two weeks ago we had a five-hour shutdown to cut over from the old core data center switch to the new one. ...That would be five hours of work at the end of days of planning and fact-gathering.

Today we had to throw that old switch -- a vintage 2001 Cisco Catalyst 6506 -- back into service on the spot.

The results are pretty damn ugly, but we're back up. We had about half of the data center back on-line in roughly 90 minutes; the rest took another hour to restore.

We're working on recovering the config of the new switch with the loaner CPU card, and expecting a replacement for the dead card tomorrow morning. I expect we may cut back over to the new switch some time later this week.
Tags: geeking, outage, work

  • Saved by the Dell

    In the past couple of years Dell made sealed keyboards standard on the Latitude line. This makes them very spill resistant, as I discovered last…

  • Getting back into the old gig

    My process of getting my head back into my old job took a couple steps forward today. I finally picked up the stack of telecom bills that had been…

  • Geeking along...

    Poking at several free-ware / share-ware network mapping tools tonight. CartoReso is a loss LanTopolog is at a loss with large switches and ring…

  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.