netcurmudgeon (netcurmudgeon) wrote,

  • Mood:
  • Music:
Well, this has been fun. Tuesday morning I got out of the meeting with the notauditors, but not in a way that comes remotely close to good. By around ten the network was starting to come apart in ways that were not good. Ok, it's never good when you start losing chunks of your network, but when you live in a very poor city with very crappy power service in some areas you quickly sort out chunks and chunks. To wit, when the chunk that's down is up in the North end, that's bad but it's also par for the course. When the chunk that's starting to unravel in front of you includes your core routers at the point where everything is joined to everything else, that's bad.

This was one of those weird problems with things seeming to break and then fix themselves, and then break again. Pretty quickly I starting thinking "the network is acting like there's a [massive traffic spewing] virus". That triggered my doctrine of geometrically more drastic responses. (This doctrine says, try something targeted and measured to stop the problem, then move rapidly to bigger and more drastic steps until the situation is stabilized. It's a "the good of the many outweigh the good of the few" strategy. If I have to shut off a ring of seven sites in order to stabilize the network across the other sixty-plus, so be it. Once the most people are running, then you can go back and start narrowing down the problem in the affected area.) We were without Internet for a while while I got a handle on the source of the problem: multiple PCs participating in a SYN flood DDOS attack on some IRC server in Germany (network sniffers tell all).

By mid-day Tuesday we had yanked one PC in Central Office that was spewing thousands of bogus SYNs each second. I had also dropped in a filter in the router just inside our PIX firewall that screened the PIX from these millions of bogus packets, thereby stabilizing the PIX and restoring Internet access. By the end of the day I had figured out that another PC was spewing away at Fisher elementary -- I locked that down and sent one of my network engineers out to yank it Wednesday morning.

The desktop techs who checked over the two PCs found multiple virii, Trojans, and spyware programs on them. We'll probably never know which particular piece of malware turned the PCs into zombie foot soldiers in someone's DDOS network. At least things are quieting down. I was out today and I need to catch up with my troops tomorrow to see what generated some NMS alerts today. It never ceases to amaze me how much damage a handful of infected PCs can do to a network.

  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.