Yesterday someone broke the network at King school. Kevin, one of my engineers, went out and discovered that no one could see our DHCP server, and that as workstation's leases were aging out or they were rebooted, they were effectively falling off the network. No DHCP = no IP address. No IP address = no work.
I can't recall a single time that we've had a problem with the DHCP server not issuing addresses, and the DHCP request forwarding mechanism in the routers (that forwards a DHCP request from a PC on to the central DHCP server) has never spontaneously "unconfigured" itself. The symptom usually means that someone has setup a rogue DHCP server in the building. But that's usually signaled by people getting bogus addresses (a la 192.168.0.xx), not no addresses. It took a little while for the true nature of the problem to become clear.
After about twenty minutes on site, Kevin's laptop got an address: 192.168.110.xx. Ah ha! Part of the information your PC gets from the DHCP server is the IP address of the DHCP server. From that you can find the MAC address of the server's network interface, and from that you can look into the network switches and identify exactly which port the server is on. With the switch port identified, you trace the patch cable to a port in the patch panel labeled with the room number and jack number.
The culprit? As suspected, a crappy Linksys wireless AP / firewall router. Why were people getting no IPs? There are over a hundred PCs in the school, and the Linksys box probably only has a DHCP scope with 16, 32, or 64 addresses defined in it. All the addresses were taken up, hence the rest of the people were getting no address. It was random chance that when an address freed up in the box's scope, Kevin's laptop got it. We would have found the problem eventually, but this lucky break sped things up considerably.
Kevin, at my direction, went to the classroom with the rogue AP and seized it. There was no one there, so he left a note directing the teacher to contact me about the problem he or she had caused on the school's network. He then headed off to his next assignment.
Minutes later my phone practically leapt off the desk. Ms. L. was on the line, and she was on a tear. The facts that her actions were a) a violation of published policy and b) responsible for causing the entire data network at her school to grind to a halt meant nothing to her. We had taken her thing and she wanted it back. And she wasn't going to let me get a word in edge-wise while she was ranting, either. I can't recall the last time I had to shout someone down on the phone. I'm still so mad that I could chew nails. What on Earth happened to people having at least an iota of contrition when they f*ck things up?
If anyone has an antidote for this epidemic of entitlement, send me a case, will ya? Thanks.