?

Log in

No account? Create an account

Previous Entry | Next Entry

Twenty two and a half hours

That's how long the MS Exchange repair utility took to run through my department's email store. Starting at 4:00 PM yesterday and running straight through to 2:30 PM today. To say that the people in the department were unhappy with the lack of email (and the lack of connectivity for Blackberry users enterprise-wide) is a substantial understatement.

It goes something like this. Late yesterday morning one of our Exchange servers was infected with a worm (despite being up to date on security patches and having current McAfee anti-virus .dat files). The worm apparently modified the McAfee "do not scan" list (trying to avoid detection, most likely).

Well, modding the "do not scan" list had the effect of removing the Exchange server's database files from the exclusion list. McAfee started scanning the database files as Exchange was running, and soon enough found something it didn't like. It "cleaned" the database transaction log file for our message store and *BLAMMO* Exchange had a grande mal seizure.

My crew repaired the smaller mailbox stores in the message store first, leaving the 41 GB MHIS message store for last. By 16:00 yesterday they started eseutil on our mailbox store and waited. And waited. And waited. And waited.

Microsoft's documentation suggested that, for Exchange 2003, eseutil would run roughly an hour for each 10 GB of mailbox store. We quite reasonbly through that the tool would finish by late evening. By morning it was sitting at the last stage "deleting unicode cleanup table" (WTF? That's about as informative as "PCLOADLETTER" from Office Space)

There it sat. ALL. DAY. LONG. At the twenty-two-and-a-half hour mark *BING* it was done. Exchange mounted the mailbox store, email started flowing, and life returned to normal.

The fallout will start falling on Monday. As surely as the sun will rise.

Comments

( 3 comments — Leave a comment )
also_huey
Mar. 6th, 2010 02:32 am (UTC)
My condolences.

The thing I hate the most about Exchange is this: even the most hardened of anti-Microsoft zealots (of which I am not one) are forced to admit that it's a necessary piece of the enterprise IT. The suits are so impressed with the pretty Outlook interface: shared calendars, delegation, hooks to Project and AD, blah blah yackety smackety -- and all that stuff is nice, really -- while at the backend, there's this big ugly message store that takes eons to do anything with. Behind the shiny prettiness lies a big steaming turd sandwich that somebody has to spend their nights and weekends eating. It's a catastrophically bad product, and yet, practically every IT staff is forced to run it.
netcurmudgeon
Mar. 7th, 2010 02:18 am (UTC)
Amen, brother.
thecoughlin
Mar. 7th, 2010 03:35 pm (UTC)
my condolences on the lost hours of your life on this one....
( 3 comments — Leave a comment )

Latest Month

January 2017
S M T W T F S
1234567
891011121314
15161718192021
22232425262728
293031    

Tags

Powered by LiveJournal.com
Designed by Lilia Ahner