Man, this has not been a good week for my network:
ThursdayNeo over at the HPL nailed up a bridging loop that dropped their entire subnet and wiped out domain authentication enterprise-wide for fifty minutes, and left me chasing errors clear into Monday...
Monday morning schools in the North end couldn't get to voicemail, and a couple couldn't make or receive outside calls. In a VoIP network, this means a routing problem -- which I caused by resetting HPL's ATM switch early Sunday to clear one of the leftover problems that Neo caused with his fricking loop. We got things put right by 8:30 AM.
Tuesday the PIX firewall that handles our connection with CEN (the Sate education network) locked up, halting all school and library Internet traffic for another fifty minute span.
Wednesday morning CEN puked -- they spun up two new Cisco Content Engines (web caching servers) to augment their existing pair, and promptly trashed all the web traffic going through their network. Not my problem to fix, but still a problem for my users, and it would be demanding a sophistication they don't possess to get them to differentiate between my network failing them and CEN failing them.
Wednesday afternoon someone at Weaver High School with a $20 hub plugged two cables into the wall and created a loop that knocked the whole school out for an hour. Obviously we need reevaluate not running Spanning Tree, but we haven't had one of these in three years and now we get two in a week -- WTF?
TODAY, the CEN PIX locked up again. This time it was at 6:55 PM which meant that I had to go into Hartford to resuscitate the damn box. Total outage time was less than 30 minutes, but damn, we can't catch a break this week.
So, there you have it. Six major, service-disrupting outages in eight days. If you pray, and have some juice with the almighty, pray for a quiet Friday. Thank you.