?

Log in

No account? Create an account

Previous Entry | Next Entry

Well, this has been fun. Tuesday morning I got out of the meeting with the notauditors, but not in a way that comes remotely close to good. By around ten the network was starting to come apart in ways that were not good. Ok, it's never good when you start losing chunks of your network, but when you live in a very poor city with very crappy power service in some areas you quickly sort out chunks and chunks. To wit, when the chunk that's down is up in the North end, that's bad but it's also par for the course. When the chunk that's starting to unravel in front of you includes your core routers at the point where everything is joined to everything else, that's bad.

This was one of those weird problems with things seeming to break and then fix themselves, and then break again. Pretty quickly I starting thinking "the network is acting like there's a [massive traffic spewing] virus". That triggered my doctrine of geometrically more drastic responses. (This doctrine says, try something targeted and measured to stop the problem, then move rapidly to bigger and more drastic steps until the situation is stabilized. It's a "the good of the many outweigh the good of the few" strategy. If I have to shut off a ring of seven sites in order to stabilize the network across the other sixty-plus, so be it. Once the most people are running, then you can go back and start narrowing down the problem in the affected area.) We were without Internet for a while while I got a handle on the source of the problem: multiple PCs participating in a SYN flood DDOS attack on some IRC server in Germany (network sniffers tell all).

By mid-day Tuesday we had yanked one PC in Central Office that was spewing thousands of bogus SYNs each second. I had also dropped in a filter in the router just inside our PIX firewall that screened the PIX from these millions of bogus packets, thereby stabilizing the PIX and restoring Internet access. By the end of the day I had figured out that another PC was spewing away at Fisher elementary -- I locked that down and sent one of my network engineers out to yank it Wednesday morning.

The desktop techs who checked over the two PCs found multiple virii, Trojans, and spyware programs on them. We'll probably never know which particular piece of malware turned the PCs into zombie foot soldiers in someone's DDOS network. At least things are quieting down. I was out today and I need to catch up with my troops tomorrow to see what generated some NMS alerts today. It never ceases to amaze me how much damage a handful of infected PCs can do to a network.

Latest Month

January 2017
S M T W T F S
1234567
891011121314
15161718192021
22232425262728
293031    

Tags

Powered by LiveJournal.com
Designed by Lilia Ahner