Net.Curmudgeon's Journal

First day back at work...

2010-01-06T04:06:14Z

...since the arrival of Little Guy #2. Oy. The Boss' Exec wasn't kidding -- my in-box was towering life Everest. It also felt like I was getting hit with two week's worth of backlogged questions from my crew. By the end of the day I was wiped out.

Coming home was pretty good. ashacat's mom had just left, so Asha had had help throughout much of the day. Both boys were in good shape -- #1 was contentedly watching heavy equipment on YouTube (with Mom driving the laptop) and #2 was asleep. Ah, domestic bliss. :-)

Securitize yourself...

2009-10-28T00:57:16Z

So this is what I did with the second hour of my morning:

The inside of the doors on our main storage area

Tie-down rings, bolted through the doors, with chain and padlock. We'll keep it locked from the inside from now on; opening it only for taking in large deliveries.

Not to shabby on the outside, either. Those four round heads are the bolts anchoring the tie-down rings. The anti-tamper plate over the latch didn't do a bit of good -- the would-be crook just bent it backward to get at the latch.

Sometimes...

2009-10-27T02:50:07Z

...it seems like you just do everything wrong.

At least, the cops certainly thought so.

Rewind.

Friday afternoon 4:45 one of my people calls me. She has just confronted -- and scared off -- a man who broke into our storage room, and then into her office. I determined that she was unhurt and told her to call security. Then, I went outside to get something done before the weather closed in.

Security came down. They called building maintenance to put the doors back together. My employee did not call the police. Security did not call the police. *I* did not call the police.

I checked our network cameras when I got in and found -- to my deep dismay -- that the camera covering the storage room door had been out of commission since the 8th of September.

I went in Friday night to fix the camera and make sure that the space was secure.

Today -- this morning -- I call the cops. An hour or so later an officer showed up. We told the story. My employee described the break-in and gave a description of the intruder. And then the lecturing started.

The officer worked us -- but mostly my employee -- over for not immediately calling 911. And he was right. The perp could have been high, or nuts, or not an easily scared man looking to do a property crime. We could have come in Saturday morning (major weekend upgrade -- a long story for another day) and found a corpse. Then the shift supervisor -- a sergeant -- gave us the 911 lecture. Then one of the detectives from ESU (the Evedentiary Services Unit) gave the same lesson. At least he had some compassion -- he listened to my employee and said that "It happens -- you get all shaken up. Next time, call us, that's what we're here for."

They dusted for prints, and collected the chunk of metal we believe the intruder was using to jimmy the doors. Hopefully tomorrow the building will give us a DVD from the one camera that might have caught something.

The cops agreed with what has been our long-standing assessment: the building's security is for shit, and the landlord's handling of incidents is half-assed at best.

Damn, just damn.

Almost only counts in horse shoes and hand grenads...

2009-10-16T21:17:48Z

This morning we were helping a trucker unload roughly three tons of gear in the street and move it into our building. Some archetectural genius designed our loading dock to be too low to accommodate anything taller than a big FedEx van. So, truckers have to use their pallet jacks to move goods onto the lift-gate at the end of their truck (presuming the shipper sent a truck with a lift gate), lower the gate to the street, and then we help push the pallet up the ramp into the parking garage and across into our area.

Today the trucker has an accident with a pallet of six large UPS units. As he lowered his lift gate, the pallet leaned over and then fell off. The equipment was trashed, but no one was hurt -- just. This is my note to our rep at the firm that sent the goods:

[Dear rep];

We need to come up with a better way to ship large orders. As the past several years have demonstrated, the common carriers are either unable or unwilling to meet out requests for trucks able to fit into the 260 Constitution Plaza loading dock. Not only are we rather tired of having to move thousands of pounds of merchandise off of the street into the building, but sooner or later SOMEONE IS GOING TO GET HURT.

Sooner or later almost came today -- twice. [One of my employees] was nearly struck by an inattentive motorist. The trucker's helper came close to getting crushed when he tried to brace-up that pallet of UPSs as it toppled off of the truck's lift-gate. That was eight hundred and four pounds, plus a pallet and pallet-jack falling to the ground. He would have been seriously injured -- and perhaps killed -- had my men not yelled to warn him off.

So, I await your response. We will do all that we can to work with you to come up with a better -- safer -- solution for shipping our large orders.

Best regards,

[Me]

Columbus day...

2009-10-13T00:37:02Z

...made for a delightfully quiet day at work. The City half of the department had the day off, and many of the BOE half of the department took the day off. So, there were all of eight of us in the building. It was wonderful. Not that I don't have great people, mind. But it was a treat to work a whole day with almost no interruptions.

Old Hoss

2009-10-06T12:14:52Z

Late last week I virtualized the last of our Pentium II servers. This particular box had been working as an external DNS server for years. I think the last time it went down was due to a power failure in the computer room.

Last login: Fri Oct 2 13:18:37 2009 from 209-164-157-64.hartford.gov
Linux 2.2.19.
root:~# uname -a
Linux ns4 2.2.19 #2 SMP Fri Aug 1 13:20:47 EDT 2003 i686 unknown
root:~# uptime
2:09pm up 464 days, 1:43, 2 users, load average: 0.00, 0.00, 0.00
root:~# shutdown -h now

Broadcast message from root (ttyp1) Fri Oct 2 14:11:01 2009...

The system is going down for system halt NOW !!
root:~#

...An old Dell PowerEdge 4300 with dual 400 MHz Pentium IIs and maybe 512 MB of RAM. It never ceases to please me to think about how much useful work you can get done when you're not wasting a billion (or two) clock cycles each second being pretty. Stuff this in Aero Glass's pipe and smoke it.

I can has iSCSI

2009-10-04T20:52:14Z

I think that I may actually have a handle on this iSCSI thing. I have successfully configured an iSCSI target (the server that provides the disk space) on Linux (Slackware 12) and iSCSI initiators (the clients that use the disk space) on Windows XP and Linux (Slack 12, again).

This is all due to the "...impending storage explosion at work." Next up will be doing some test runs in a near-production environment in the office. This could be the answer to a lot of our troubles.

Wiktory!

2009-10-04T04:16:38Z

I have beaten iSCSI on Slackware 12 into near submission. Now I can go to bed.

We are facing an impending storage explosion at work

2009-10-03T00:46:02Z

The tidal wave of schools wanting home drives and email accounts for their students is about to break over our heads. I have been talking options with my team -- our discussions have ranged from high-end solutions to Frankenstorage to everything in between.

It just amazes me what the major storage vendors charge for their products. EMC is the highest-of-the-high-end, with NetApp (nee Network Appliance) between EMC and the likes of Dell and Sun. We have a NetApp SAN, with several shelves of 10K RPM 300 GB FibreChannel drives and a single shelf of SATA drives (for disk-to-disk-to-tape backups). A single TB of usable space cost us $9,460 with the FC shelves, and $4,892 for SATA. (That was after the super-aggressive introductory pricing wore off.) The kicker is that these costs are based on adding to our existing storage controllers. Adding a shelf once the capacity of the controllers is reached puts you into a whole new world of pain. Think $40K per usable TB.

I don't mean to offend the NetApp folks, but there's nothing overwhelmingly special about these disks. They're made by the same companies that make the disks that you and I buy. Yes, there are all sorts of wonderful features in the system ... but they'll all cost you extra to "unlock" and to maintain. It almost seems like the five star hotel effect. It's a five star hotel so you expect to pay a lot, even if the breakfast buffet isn't any different than the one at the Marriott down the block.

We've looked a assembling our own kit from commodity pieces, but there are always questions. Who will support the hardware? is a big one. BackBlaze -- an on-line backup vendor that needs staggeringly vast tracts of space -- has developed a nifty commodity box that holds 65 TB of disk in 5 rack units. Very attractive, until you understand that their givens are not your givens. It's the Google problem -- they too operate on dirt-cheap commodity hardware, and they too created their entire operating environment from the ground up. The designed each cheap server node to be expendable. In essence, they have data redundancy at the service level so they don't have to pay for redundancy at the device level.

What's a medium-ish sized IT shop to do? Piecing together an infrastructure from parts may work for a startup, but we can't do that here. Windows servers don't fit the Google/BackBlaze model, and "enterprise class" storage from EMC or NetApp will break the bank. The answer may be a middle path. Taking a good Dell server (R710, loaded with RAM, a 10Gb Ethernet card and a monster RAID card) and pairing it with one of their external drive enclosures yields costs in the range of $4,400/TB for high-speed storage and $1,300/TB for archival or D2D2T storage ... even with the gold-plated 3-year 24x7 4 hour response support plan. Linux is a perfectly mature iSCSI target / NFS server, so I know that we can make it work.

Film at 11. We'll see how this turns out.

Four random things make a post

2009-10-02T01:10:15Z

We finally knuckled under and turned on the heat this afternoon. The house was 60° this morning and just did not warm up.

ashacat has a nasty cold, so I got to take the little guy for his first visit to the dentist (he's 21 months tomorrow). We had a good time ... he got to sit on dad in the chair, and look at all the stuff, and play with the little water spigot and cup. He wasn't up for showing the doc his teeth, but that will come. Next time I go in for a cleaning (February) I'll take him and he'll get to see that. The dentist's objective (and ours) is to get him familiar and comfortable with the office over time, so that the first time he goes in for real some of the fear factor won't be there.

My hairline has been in full retreat for two years now. Its effect was made clear to me the other day when a trucker, describing me to our front desk person, said "a red haired guy about forty". And there you have it, Mr. 'I thought you were in your twenties' is finally looking his age. :-)

ashacat has completed chapter 5 of 10 of the draft of her dissertation. Half way in terms of chapters, but probably closer to 3/4 in terms of page count. You go girl!

As previously noted...

2009-09-29T01:38:43Z

My work life and my home life seem to often run in parallel streams. Today at work I was Mr. manual labor, shuffling boxes, moving three dozen PCs, helping to push 30 new servers up the ramp into our building. Tonight at home it was carpentry, carpentry, carpentry -- building new shelves to help us get the basement organized.

Hoot!

2009-09-27T23:29:39Z

...Ran the 3rd installment of my Interstellar Marshals Service game today for ctwriter, kriz1818 and taichigeek. evynrude couldn't make the game, so her character became another victim in the kidnapping plot (an old GM's trope, I know, but one that worked well).

The players did a great job sifting through mountains of evidence, cracking the case, rescuing the hostages, and unearthing evidence of a conspiracy penetrating the ranks of the Marshals Service and the Navy.

Yeah. All that and style too.

Some people's families...

2009-09-23T04:32:50Z

For most people, when their parents go away on vacation, they ask them to take care of the dog or water the plants. My dad just asked me to close a grave.

...Ash interment at the Old Church Cemetery, Saturday morning 9:00 AM. The good news is that the hole required for cremains is really rather small, and the 2x good news is that the minister conducting the service has a practice of asking the friends and family present to each put a shovel full of dirt in. So, to quote my dad "it'll probably be almost all filled in by the time they're done."

Eh, what can you do? Besides, some of my best story-telling stories have come out of that graveyard.

PS: I am amused, and a little weirded-out by the fact that Firefox's spell-checker knows the word cremains.

Birthday wishes...

2009-09-22T20:11:50Z

HAPPY BIRTYDAY
matociquala!

PVC observation...

2009-09-22T01:13:01Z

matociquala and avynrude were visting ashacat and me on Sunday. It being a lovely day, we were enjoying the blue sky and sun out on our deck. Whereupon we learned the answer to this question: What is the safe operating lifetime of a good quality PVC lawn chair?

The answer is twelve years. It is most decidedly not thirteen.

...Two developed spontaneous stress cracks, and Bear's decided that enough was enough and died -- rather explosively.

The President of the United States of America...

2009-09-10T00:20:04Z

I am getting ready to listen to President Barack Obama for the second time today...

I watched a video of his school speech this afternoon. And, as expected, despite all of the far-right looneys' rantings, I did not detect an iota of socialist propaganda. I did detect a lot of sound concepts and good leadership. So, it is a keen pleasure -- not felt at all over the past eight years -- to await a Presidential address.

YES WE CAN

Sometimes you have to trust your gut...

2009-09-07T23:35:04Z

Thursday evening I was at Hall's Arrow with ashacat and matociquala for our usual weekly archery outing. The ladies had both recently had work done on their bows by the guys at Halls. My bow had been feeling a little odd, in an unquantifiable way recently. The serving on the bow string was starting to get worn, so I decided "When in Rome..." and trooped over to the counter and asked then to have a look.

The guys said "yes, you need a new serving" and started unwinding the old one from the bow string. Then we all got a surprise. About 1/3 of the strands in the string were broken right below where the knock had been crimped on. Righty-o then.

One new bow string later my bow was back in shooting shape, feeling good, and making its normal noises.

The moral of this story? When your gut tells you something isn't right with a machine you know very well listen.

OK all you smart people...

2009-09-07T23:23:58Z

View Poll: Croco-nomia

Do I get points for not killing him?

2009-08-21T23:38:28Z

Last week (during the BTOP death-march), one of my coworkers -- the GIS manager, on whose good work much of our application depended -- came to me with the complaint that when he and his staff person opened a project in ArcMAP for the first time each day it was slow as ~~hell~~ molasses, but that subsequent opens were quick as usual. The problem started three weeks ago says he, and he intimated that it might be a "local network problem" because none of their users elsewhere were complaining... *

So, over the course of a week we spent a bunch of time testing and troubleshooting. The good news is that the problem was generally reproducible, but our testing results were just ... odd. And, they sort of supported the idea of a network-level problem. However, my gut was telling me that it was a client or server issue.

The more I dug into the network, the stronger my hunch got. Thanks to many dinner-table conversations with taichigeek, I have some knowledge of how the inner workings of a database server, well, work. Yesterday afternoon I had my colleague and his minion perform the following test: go into ArcMAP and wait through the very long project open, then reboot and immediately dive right back into the same ArcMAP project. The result? Prior to previous tests where the interval between rebooting and starting ArcMAP was a lot longer,** this time they were able to jump into ArcMAP and load their projects at "normal" speed. Ah ha!

That result told me that the problem was not client side. The arrows were now pointing firmly at their ArcGIS database server.

This box is a hefty machine -- twin dual-core 3GHz Xeon (Dempsy) processors with 8GB RAM and 1.4 TB of usable disk -- running a 400GB+ database packed with geo-coded data and high-resolution 'pictometry' (very sharp aerial photos). I spent some time this morning staring at perfmon and rummaging through the event logs. I should have started here a week ago.

The event log was full of entries about SQL server timeouts trying to get buffers and latches, and very long instances (e.g. 680,000 ms - ~11 minutes) of trying to grow the transaction log for the GIS database. The machine needed the services of a competent DBA.

After lunch I went to pay my coworker a visit. His minion greeted me with "[He] has something to fess up to." Oh? He told me that mid-morning he had decided to take a look at the SQL database, and discovered that it had been a wee bit longer than he had thought since the last time he purged the database transaction log. How big was the log? Three hundred eighty gigabytes. He rather circumspectly admitted that their applications had been running fine ever since he pruned the log.

He also told me that he had called ESRI support and found out how to configure the transaction log so that it would never grow beyond 2GB (old entries would just be pushed out). And that is probably why I didn't kill him.

* In the final analysis this probably has a lot more to do with our to GIS people starting their days at 7:30 AM and most other city workers starting at 8:30 or 9:00.
** Due to things like shifting subnets, moving connections from one switch to another, etc.

Done ... finally

2009-08-16T00:45:56Z

After a month of nearly continuous work, our grant application to the Broadband Technology Opportunity Program is in. We were scrambling like mad things at the end to get it in on time ... then due to the on-line application web site crashing at least twice in the last 36 hours before deadline, the granting agencies moved the deadline back six days. Good that the feds did that, frustrating for those of us busting our humps to be done on-time-if-not-a-day-early.

Of course:

I have no idea when the planned award date is
This pile of work is only to get through Step 1 of the process. If we get to Step 2 we can expect a round of Q&A with NTIA
This would have been pure, geeky fun if it weren't for the tremendous time pressure
I am so completely brain fried
I need to catch up on missed time with ashacat and our son

My life...

2009-08-06T00:15:54Z

...for the past three weeks, has been eaten by BTOP -- The Broadband Technology Opportunity Program grant. Hartford is applying. Time from the notification of funding availability (publication of the 'NOFA') and the due date? Four weeks. So, we're compressing a six to twelve month design-engineering-political process into just days.

Part of our application will have to do with demonstrating that we have underserved population. Another part will be asking for a waiver of the grant's 20% local match requirement. Both of which hinge on clearly explaining the depths of Hartford's poverty. I sent this to the Boss this evening:

Attached is an extract of our E-Rate demographic data. It doesn't demonstrate under-served directly, but it certainly establishes impoverishment.

As a school district, 80.30% of Hartford's students qualify for free or reduced-price lunch under National School Lunch Program guidelines. From an E-Rate perspective this translates into an 86% discount rate. This marks us as poorer than Waterbury (85% discount), and it stands in stark contrast to the discount rates of our surrnounding suburbs: Avon (40%), Glastonbury (39%) and Simsbury (40%). By the USDA's rubric, 4 out of 5 students in Hartford's schools are poor enough to need assistance to get lunch.

This, however, is not the entire story. Hartford's poverty -- as measured by NSLP eligibility -- is diluted by the district's success in drawing suburban students into it's interdistrict magnet schools. If we set aside the NSLP data for these five magnet schools, a grimmer picture emerges. Looking at the 35 schools that only enroll Hartford students, 84.82% of those students qualify under NSLP. The mean qualification level for these schools is 86.20%, with a median of 93.32%. Put simply, in half of Hartford's schools, more than 9 out of 10 children come from impoverished families.

Next time you're on the street, look left, look right, and imagine that only one out of every ten people you see comes from a family with enough money to buy a goddamn hot lunch.

Oh yeah, and contrary to most of human history, summer is a hungry time in Hartford. Schools are closed so many children have no place to get breakfast, let alone lunch.

I feel a real rant coming on, so I'll stop now. The figures speak for themselves.

Birthday wishes...

2009-07-23T02:09:00Z

Happy birthday to britgeekgrrl!!

May the next year be easier than your last.

So, of course...

2009-07-22T02:30:43Z

I shut one of my machines down tonight to have a look-see at its innards. I did my looking around, then started the box back up. The RAID card starts wailing. Poke, poke, poke So, of course one of the (admittedly older) SCSI drives has taken this opportunity to cease to be. It's expired and gone to meet its maker! It's a stiff! Bereft of life, it rests in peace! If it wasn't screwed to the drive tray, it'd be pushing up the daisies! It's metabolic processes are now history! It's off the twig! It's kicked the bucket, it's shuffled off this mortal coil, run down the curtain and joined the bleedin' choir invisible!!

But that's what we have RAID volumes for.

Hurm...

2009-07-19T23:30:47Z

I have been working my way through the circuits in the house doing remediation on the Aluminum wire to copper wire connections. I have learned several things so far:

The electricians who wired this house in the first place were, without a doubt, real pros. Every place where wires are joined with a wire nut, the work is so tight and consistent it looks like it was done by a machine.
The outfit the previous owner had in to reterminate all of the switches and receptacles were a bunch of mooks who should just turn in their tool belts. They, if they had ever read the Story of Mel, certainly did not realize that it applied to them and this job. One of the primary rules I work by is don't make things worse. They made things worse. Potentially a lot worse. They made every mistake the CPSC warns you about when remediating Aluminum wiring, plus a big, basic electrical error. When the breaker in the panel says 20 Amps, you don't put 15 Amp receptacles on the circuit.
Always, always, always test each outlet with your voltage finder before laying hands on it. Even if you're sure it's on the same circuit you were just working on somewhere else. Additional circuits can appear in a wall box (found that one today), or you can forget that you re-energized the circuit for testing right before you took a break (found that one the hard way a month ago). Thankfully Mr. Zappy isn't all that aggressive at 120 volts across dry skin...

That's all from home electrical land.

Birthday wishes...

2009-07-19T22:57:43Z

Happy birthday thecoughlin!!