My year of Hell in 4 days


There is a Star Trek Voyager episode entitled “Year of Hell”. Its about a span of 365 days in the life of the crew where pretty much anything that could go wrong, goes go wrong and I think they barely survive (its been a while since I’ve seen it). Well, over the last 4 days, I’ve had my year of Hell starting from Friday night to yesterday around 6:30pm. I’ll summarize:

1. Friday night, I installed DST patches

a. DST patch caused a problem where my info stores would not
mount in Exchange 2003
b. The DST patches also broke the goodlink system for mobile e-mail
c. Turned out that MS had changed the way the permissions work in
Exchange and AD and we had to re-set the permissions on our
GoodAdmin account
c. Spent up until 11:30pm working on the DST patches and fixing the
goodlink account while on the phone with Good support

2. Saturday – all day

a. Checked the server from previous night to ensure everything is
still working
b. Found everything to still not be working.
c. Found some objects in Exchange 2003 were not
inherriting permissions correctly. Had to continue
resetting permissions for those items and testing.
d. Continued working with Good support and testing and finally got the
Goodlink mobile e-mail working
e. Had to follow a workaround by allowing inherritance on the
AdminSDHolder AD object and allowing time for replication
f. Spent the rest of the day testing the mobile e-mail system paranoid
that it would fail again.
g. Abby got sick Friday night, so I got maybe 2 hours of sleep Friday
night and Sat Morning

3. Sunday – all morning

a. Continued testing the mobile e-mail system, all tests were working
ok, external and internal.
b. I stayed home with Abby all day to take care of her, she was really
sick and had a very high fever.
c. Got my first almost full night of sleep, since Liz was kind enough to
stay in the living room with Abby

4. Monday – All DAY!!!

a. Started getting complaints that people couldn’t open some e-mail
messages in Outlook
b. Got some reports that people were missing e-mails, that the volume
of messages received over the weekend and morning were very low
c. To make it more frustrating, all the external tests and internal tests
I was doing were working fine
d. Checked the mail queues in Exchange 2003 and found hundreds of
queued messages, they were stuck in the MTA queue
e. Nothing I tried would release the messages
f. I called Microsoft and paid my $245 fee for support, and spent about
8 hours on the phone with 2 separate Microsoft engineers.
g. Once I got through to the escalation engineer, we got the issue fixed
within 3 hours or so.
h. Turned out that MS had made two code changes to alter the way
Exchange does recipient and sender address formatting or lookups.
i. The quick fix was to uninstall patch 930241. So I did, and that
restored functionality completely, for both symptoms.
j. I also worked with MS to run some store traces to try to
identify the issue.
k. After some minor permissions adjustments for two servers who were
still not inherriting permissions correctly, everything started to
work normally.
l. I did NOT have lunch this day, nor anything to drink. I was on the
phone all day!
m. At 6:35pm I finally left the office and went home and crashed!

So as you can see, thanks to Congress for screwing around with the Daylight saving configuration and thanks to Microsoft for making some code changes that had unexpected side effects, I ended up having a very bad few days indeed. I really didn’t get a weekend, no break, barely any sleep. Its going to take me a while to recover from that. I need a vacation! Sarah has off all next week for spring break, I’m thinking of asking for next Wednesday off to break up the work week and give myself a break.

Advertisements

About Joe

I am the author of this blog, IT engineer, husband, father, and somewhat of a nerd.

Posted on March 12, 2007, in Professional/Tech and tagged , , , , , , , , , , , , , , , , , , , , , , , . Bookmark the permalink. 4 Comments.

  1. P.S. I plan to post the specifics with all the details, microsoft KB articles and what we found out afterwards. I’m working with Microsoft now to do some issue reproduction in a test environment to help solve the issues we experienced. Im typing all that up now.

  2. I feel for you man. It is times like these that I am glad we do not use Exchange!

  3. Update: Microsoft has released a post SP2 update for Exchange 2003. They have also accepted a backport for other patch versions, so there will be a hotfix for other patch versions of Exchange 2003 as well. I think all the patches will be available on the Microsoft website sometime in the next week or two.

  4. FYI – a bit late to post this, but MS released a new KB based on tracing obtained from my server. The new KB is at http://support.microsoft.com/kb/926666 And I’m happy to say it should fix all the problems mentioned in my previous posts. So if you are having this issue, check out MS KB 926666!

%d bloggers like this: