netcurmudgeon (netcurmudgeon) wrote,

  • Mood:
  • Music:

First day back from vacation...

So, my first day back at work from vacation was ... a bit of a zoo.

The morning was spent on paperwork, finances, finding out who exactly got laid off last week, and more paperwork.

The afternoon was spent dealing with the worst network failure we've had since January of '06.

Our brand new Nortel ERS8306 routing switch -- the core of our data center, in service for only two weeks, but after a month of burn-in and bench time -- failed catastrophically at 13:24.

The new switch -- brought back from the dead with a loaner CPU card from Nortel

Two weeks ago we had a five-hour shutdown to cut over from the old core data center switch to the new one. ...That would be five hours of work at the end of days of planning and fact-gathering.

Today we had to throw that old switch -- a vintage 2001 Cisco Catalyst 6506 -- back into service on the spot.

The results are pretty damn ugly, but we're back up. We had about half of the data center back on-line in roughly 90 minutes; the rest took another hour to restore.

We're working on recovering the config of the new switch with the loaner CPU card, and expecting a replacement for the dead card tomorrow morning. I expect we may cut back over to the new switch some time later this week.
Tags: geeking, outage, work

  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.