This was one of those weird problems with things seeming to break and then fix themselves, and then break again. Pretty quickly I starting thinking "the network is acting like there's a [massive traffic spewing] virus". That triggered my doctrine of geometrically more drastic responses. (This doctrine says, try something targeted and measured to stop the problem, then move rapidly to bigger and more drastic steps until the situation is stabilized. It's a "the good of the many outweigh the good of the few" strategy. If I have to shut off a ring of seven sites in order to stabilize the network across the other sixty-plus, so be it. Once the most people are running, then you can go back and start narrowing down the problem in the affected area.) We were without Internet for a while while I got a handle on the source of the problem: multiple PCs participating in a SYN flood DDOS attack on some IRC server in Germany (network sniffers tell all).
By mid-day Tuesday we had yanked one PC in Central Office that was spewing thousands of bogus SYNs each second. I had also dropped in a filter in the router just inside our PIX firewall that screened the PIX from these millions of bogus packets, thereby stabilizing the PIX and restoring Internet access. By the end of the day I had figured out that another PC was spewing away at Fisher elementary -- I locked that down and sent one of my network engineers out to yank it Wednesday morning.
The desktop techs who checked over the two PCs found multiple virii, Trojans, and spyware programs on them. We'll probably never know which particular piece of malware turned the PCs into zombie foot soldiers in someone's DDOS network. At least things are quieting down. I was out today and I need to catch up with my troops tomorrow to see what generated some NMS alerts today. It never ceases to amaze me how much damage a handful of infected PCs can do to a network.