netcurmudgeon (netcurmudgeon) wrote,


Just before lunch today I noticed that Internet access through our commercial internet feed was really sluggish. I took a look at the traffic levels reported in MRTG and saw that the traffic on this 3 Mbps link was pegged at 3 Mbps -- and had been since 8:45 in the morning. Not good. Doubly Not Good was that instead of being pegged by inbound traffic, the circuit was instead being saturated by outbound traffic. That meant that someone inside the network had been pouring a vast amount of traffic out onto the Internet for almost three hours. I resolved to check things when I got back from lunch.

The blue line is outbound, green is inbound

Traffic was still maxed when I got back at 12:30, so I grabbed my laptop and headed over to the building where our firewalls are located. I ran a couple of traces with Ethereal, and then did some quickie analysis on the captured data. The "top talker" was our Exchange server -- vomiting forth a volume of traffic that was an order of magnitude greater than the next highest machine. The vast majority of the total traffic was SMTP (email), to a number of different hosts on the 'net. My immediate suspicion was that some genius user was trying to email a DVD to a couple friends. In the end, I wasn't too far off.

Once I confirmed that the offending system was the Exchange box, I trundled off to see Mark-my-unindicted-co-conspirator. Mark started digging around in the Exchange server outbound queues, and we quickly found dozens of 26 MB messages waiting to go. A little further digging gave us the sender's ID (the public information officer for one of the City departments). Mark attached to their mailbox (it's good to be the king) and we took a peek in their sent items. Sure enough, at about 8:45 the user had sent out a video file to scores of outside civic and media people. Mark called the wayward user, left a message, and then started deleting emails out of the queues.

About twenty minutes later Mark was about done cleaning out messages when the user called back. Thankfully this person is one of the good eggs. They were properly mortified over the trouble they caused, and were sincerely apologetic. Yes, it may have only been a couple of minutes long, but that was a near broadcast quality clip you sent! Next time they'll post it to the department's web site and email a link.

Mark and I amused ourselves picturing the DSL modems of some of the little non-profits this mail went out to melting under the barrage. :-)

...By 2:30 things were back to normal.

  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.