netcurmudgeon (netcurmudgeon) wrote,
netcurmudgeon
netcurmudgeon

  • Location:
  • Mood:
  • Music:

And a happy good morning to you too!

I got up this morning around 8:30, toddled down stairs, and plunked myself on the couch for my ritual morning look at the electronic world. My inbox had a bunch of messages from the automated monitoring system (a personal version of this) that keeps watch on my various servers. From the monitor's point of view, Alpha and Beta (my two production web servers) were going up and down.

We've had a lot of rain in the past few days, so my instant suspect was a problem with the DSL circuit that alpha and beta share for their Internet access. Pings from home to the servers were dropping 50% of the time. But, as I started to dig deeper, that hypothesis started losing ground. They weren't responding to SSH, and there was an odd period of high traffic on the MRTG traces for both servers. I began to suspect that the two servers had been compromised. (In other works, hacked! cracked! pwned!)

Thankfully that turned out to be wrong as well. I finally managed to SSH into Alpha and Beta from another host. A quick check of logins, active processes, and open network ports showed that both servers were exactly as they should be.

I turned my attention back to the possibility of a network problem, but things weren't adding up. Normally, when you are having connection problems, the problem is in what telco types call the last mile or the local loop meaning that circuit at the very edge of the carrier's network to you. Yet, both my cable service and the DSL service at Alpha's and Beta's undisclosed secure location appeared to be fine. The problem only manifested itself when one site tried to talk to the other. A traceroute from here to there brought the real problem to the light:


Microsoft Windows 2000 [Version 5.00.2195]
(C) Copyright 1985-2000 Microsoft Corp.

C:\Documents and Settings\netcurmudgeon>tracert alpha.houseofhum.com

Tracing route to gpip.org [65.75.17.31]
over a maximum of 30 hops:

  1     7 ms     6 ms     8 ms  10.4.40.1
  2     7 ms     7 ms     7 ms  glstsysc01-gex0102000.ct.ri.cox.net [68.9.8.33]
  3    10 ms    11 ms    11 ms  provsysj01-atm020311.rd.ri.cox.net [68.9.10.133]
  4    10 ms    10 ms    12 ms  provdsrj01-ge600.rd.ri.cox.net [68.9.14.101]
  5    12 ms    10 ms    12 ms  provbbrj01-ge020.rd.ri.cox.net [68.1.0.48]
  6    17 ms    15 ms    15 ms  NYRKBBRJ01-so000.R2.ny.cox.net [68.1.0.51]
  7    17 ms    16 ms    16 ms  68.1.0.253
  8    22 ms    19 ms    20 ms  68.1.0.250
  9    28 ms    35 ms    24 ms  mrfdbbrj02-ge030.rd.dc.cox.net [68.1.1.3]
 10    24 ms    23 ms    23 ms  ashbbbrj01-pos020100.r2.as.cox.net [68.1.1.232]
 11    21 ms     *        *     68.1.0.220
 12    23 ms     *        *     sp0-4-ASBNVAAS.broadwing.com [206.223.115.72]
 13     *        *       25 ms  serial2-0-0.e1.nwrk.broadwing.net [216.140.9.9]
 14     *        *        *     Request timed out.
 15    28 ms    27 ms     *     p6-0.c0.nwyk.broadwing.net [216.140.17.122]
 16     *        *        *     Request timed out.
 17    25 ms    27 ms     *     65.88.47.106
 18     *       26 ms     *     hartford.atm.ntplx.net [204.213.183.9]
 19     *        *        *     Request timed out.
 20    43 ms     *        *     ip-65-75-17-31.ct.dsl.ntplx.com [65.75.17.31]
 21     *       40 ms     *     ip-65-75-17-31.ct.dsl.ntplx.com [65.75.17.31]
 22     *        *        *     Request timed out.
 23    39 ms    40 ms     *     ip-65-75-17-31.ct.dsl.ntplx.com [65.75.17.31]
 24     *        *       38 ms  ip-65-75-17-31.ct.dsl.ntplx.com [65.75.17.31]

Trace complete.

C:\Documents and Settings\netcurmudgeon>

There, right in the middle of the trip we start losing packets (the red asteriks). Further poking showed that whatever router has the IP address 68.1.0.220 was (still is) dropping half of the traffic that gets to it. The address doesn't resolve to a name, so I can't tell if it's Cox's or Broadwing's problem, but right at the border of their networks something is amis. Hopefully some groggy geek or geekette has been paged in and is looking at it.

Hey, from my perspective at least I'm not owned!
Subscribe

  • Saved by the Dell

    In the past couple of years Dell made sealed keyboards standard on the Latitude line. This makes them very spill resistant, as I discovered last…

  • Geeking along...

    Poking at several free-ware / share-ware network mapping tools tonight. CartoReso is a loss LanTopolog is at a loss with large switches and ring…

  • Progress, progress...

    Found some time today to get SpamAssassin installed on my new mail server. I did a totally default install, so we'll see what sort of tuning I have…

  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 2 comments