Log in

No account? Create an account

Previous Entry | Next Entry

First day back from vacation...

So, my first day back at work from vacation was ... a bit of a zoo.

The morning was spent on paperwork, finances, finding out who exactly got laid off last week, and more paperwork.

The afternoon was spent dealing with the worst network failure we've had since January of '06.

Our brand new Nortel ERS8306 routing switch -- the core of our data center, in service for only two weeks, but after a month of burn-in and bench time -- failed catastrophically at 13:24.

The new switch -- brought back from the dead with a loaner CPU card from Nortel

Two weeks ago we had a five-hour shutdown to cut over from the old core data center switch to the new one. ...That would be five hours of work at the end of days of planning and fact-gathering.

Today we had to throw that old switch -- a vintage 2001 Cisco Catalyst 6506 -- back into service on the spot.

The results are pretty damn ugly, but we're back up. We had about half of the data center back on-line in roughly 90 minutes; the rest took another hour to restore.

We're working on recovering the config of the new switch with the loaner CPU card, and expecting a replacement for the dead card tomorrow morning. I expect we may cut back over to the new switch some time later this week.


( 2 comments — Leave a comment )
Jul. 21st, 2009 04:02 am (UTC)
anything but nortel...
i am not surprised. Just wait until you do have the switch up for a while and begin to experience some of the nortel software bugs. We are currently in the process of pulling all of our 8300 & 8600 ERS'
Jul. 22nd, 2009 10:37 am (UTC)
Re: anything but nortel...
Everybody has bugs. Everyone has hardware problems. Last year we had a half dozen Catalyst 3560s (out of ~100) arrive with bad POE controllers...

We have had 50+ ERS8600s in production since 2005, and aside from filtering being a pain in the ass, we're very happy with them.
( 2 comments — Leave a comment )

Latest Month

January 2017


Page Summary

Powered by LiveJournal.com
Designed by Lilia Ahner