Tuesday's highlight came when one of the network's original (and still in production) core routers lost an ATM card at quarter to ten in the morning. This router handles two schools that are still ATM attached, and all of the sites that connect via leased Frame Relay circuits. It also has the router interface for the management network. I noticed that things were amiss when I couldn't get to the management servers. I traced the problem down to the router, and then to the ATM card in slot 2 of the machine. A remote restart didn't clear the problem, so I hoofed it across the street to the Board of Ed offices where the router lives.
When I got there the problem was readily apparent. Both the front (processor) card and the back (line interface) cards had red FAIL lights on. I power cycled the router, in a vain hope that cold-booting it would clear the problem. Nope. Dead card.
I powered the router down, pulled both halves of the card, and brought it back up. It booted cleanly, and I cleared the config of the ATM virtual interfaces that had been on slot 2, and rebuilt them on slot 5 (the other ATM card in the router). All-in-all, it was a nice clean failure; the thing up and died, isolating it was quick, and the recovery was painless. Much better than some godless intermittent thing.
Wednesday started with pushing numbers around for a report that City Council wants as part of their budget deliberations. Just about the time I finished, one of the phone guys knocked on my door looking for some help with connecting new voicemail servers. Halfway through pulling parts out of our spares stock I told him "you realize, we are going to take care of this right now. This is the most interesting thing to cross my desk all f*cking day!"
...I spent the next 90 minutes in the computer room over at the Board of Ed running fiber jumpers through overhead cable tray, configuring the two new switches, and generally getting things done.
In the afternoon, just as it looked like I was running out of alternatives to tackling an odious little school construction task, when Linda, one of the financial systems people, came to see me about a VPN problem one of our vendors is having. We beat the question up for a while, and it was looking more and more like an ODBC thing, not a network thing. So, I told Linda that we were going to see the Oracle. Mark-my-unindicted-co-conspirator: not the Oracle at Delphi, more the Oracle at the end of the hall. And lo, after some minutes of conversation we put on the path of righteousness. Or, at least, we had an answer that had Linda going back to the vendor with new ODBC config information and not my starting a network goose chase.
So, you know, it's all good. Of course, the Boss should have told the Mayor that he's retiring about three hours ago. I expect that we'll get the formal news tomorrow at nine AM. Good. It's both relative and transitory.
 'Card' in this case is somewhat misleading. The BLN (Backbone Link Node) router has a mid-plane architecture. Cards with processors on them slide into the front, and cards with network interfaces slide in the pack. A front card and back card work together as a unit, and are sometimes referred to collectively as a 'card', as one would with a switch or router with a backplane architecture (where a card is all in one piece and is installed as such).