Category Archives: Job related
I came across an article by Derryl Steib in the August version of Redmond Magazine. Here is a quote from his article entitled “Listen to the People you hire”.
“The bottom line is that if you don’t trust your network administrators and heed what they’re telling you, need to hire new ones”.
Thats a great quote and something many IT engineers and administrators face on a daily basis. I can’t begin to count the number of times my suggestions have fallen on the deaf ears of upper management in most of my 10 years working in IT. People like me spend a great deal of time working with the systems we are certified in and or have a great deal of experience with. I know the difference between a paper MCSE and “real” MCSE, and honestly there are a good deal of people out there that don’t know much, and admittedly there are things I don’t know that I probably should, but when it comes down to it, the best person to ask about a problem or planning for IT projects are the people that maintain/support/design/build/etc those very systems. Sales people cannot be relied upon for accurate information about products/services, and the only real source of true information about a product or system is from the administrators and engineers who actually work with those systems on a daily basis. So I reitterate what Derryl says, trust the people you have working on these systems, listen to their advice, at the very least, accept the possability that they actually know what they are talking about and know more about the subject matter than you do.
If any of you read the dilbert comic, this past Sunday’s comic was about me! Check out
This is all so true!
Posted in Job related
1. Exchange 2003 was successfully installed into existing Exchange 5.5 site.
2. IT users have been moved to the new Exchange 2003 servers.
3. Public folder permissions were modified to remove DL membership from the client permissions section to prevent ADC replication issues.
4. Exchange 2003 servers are now located in the Tampa office and in New York.
5. Work has begun on setting up Clusters for use with Exchange 2003. This involved HP DL-585s and MSA1500 SAN (up to 2.4TB)
I’ve been extremely busy latey and will post a separate item about that soon.]]>
Once again I found myself in a terrible case of IT disaster. Last Friday night I was working on some Exchange maintenance with another sysadmin from my company. We each took two of our Exchange servers and did some work. Two servers (the ones I was working with were in the US, the other two were in other countries). Once we were done, we rebooted the Exchange servers and were pleased that they seemed to come back online and mail started flowing again. However, I quickly noticed that one server in particular, (one of mine of course), was not back online yet. I couldn’t ping it, couldn’t browse to it, nothing, it was dead to the world. Desperately I started making phone calls, no one could make it to the office until the next afternoon. I tried resetting the power ports that server was supposed to be connected to on the UPS in this remote office. Nothing worked. The next day, staff from the remote office were able to determine that the problem was a fatal SCSI controller error. Aparently a memory controller board on the SCSI controller had gone bad. This card was replaced with a spare from a retired server and that fixed the issue. The server is now back online after almost 24 hours of downtime. What a mess! I was so stressed over this I could hardly stand it. I feel much better now that the server is back up!
This posting will review the approach taken in a test environment to accomplish a simulation migration of our existing NT4 domain to server 2003 with Exchange 2003. All work was done offline on a private network IP scheme on an isolated switch to prevent communication problems with the production network. Access was limited to a hardwired laptop and a test desktop. Testing was performedwith a user/mailbox on Exchange 5.5, and another user/mailbox on Exchange 2003. Migration Process
The following are the steps in order that were used to perform a test migration of the NT4 domain. Directory exports from Exchange 5.5 were used to re-create the Exchange 5.5 Directory in a test environment.
? Initial NT4 PDC was create for the domain utilizing VMWARE and was updated to SP6A. Names utilized were duplicates of production network.
? Two additional NT4 BDCs were created;one to simulate a BDC, the other was create to perform the upgrade to Server 2003 and AD. Both were installed as BDCs and were updated to SP6A.
? Three NT4 member servers were created for use with Exchange 5.5. All were installed as member servers, and updated to SP6A. Each server was named the same as in production.
? Exchange 5.5 was installed on a member NT4 server, creating a duplicate of our existing Exchange 5.5 system. Org and site names were duplicated from Production.
? Exports of the Exchange Directory were imported and were also used to create the NT4 user accounts/mailboxes used in production. This populated the NT4 user database with usernames and blank passwords. New Mailboxes were created for users on their respective servers
? A test workstation was setup on the test NT4 domain. An exchange outlook profile was created for the exchange 5.5 user.
? Tests were performed to verify that DLs and inter-org mail flow was working.
? The last NT4 BDC that was created initially was promoted to a PDC (automatically making the original PDC a BDC).
? The OS on the new PDC was upgraded to Server 2003.
? The Active Directory Installation Wizard was run to upgrade our domain to AD.
? The AD wizard installed DNS locally on the PDC, but HOSTS files were still maintained on all NT4 servers, just as in production.
? A desktop was hardwired to the isolated switch and setup as an AD-client using a user account in AD. This was to verify that AD had successfully upgraded our NT4 domain to AD.
? A new server was installed also using VMWARE using server 2003 operating system. Server was installed as a member server of the AD domain. This was for use with the migration to Exchange 2003. SP1 was installed on the server and all updates and patches were applied.
? A temporary OU was created in AD to house the objects from the ADC replication process.
? Exchange 2003 was installed on this new 2003 server, along with the ADC and SRS. This new Exchange 2003 server was joined to our existing Exchange 5.5 site.
? The ADC replicated all Exchange 5.5 objects such as DLs, custom recipients, etc, into AD. The DLs were replicated to AD as universal distribution groups.
? Testing was performed between Exchange 5.5 users and Exchange 2003 users to verify mail flow and DL functionality. All tests completed successfully. No loss of public folder permissions was experienced. Although error 9552 in the Exchange 2003 server event log was experienced. But no loss of permissions was observed.
The above information was a test of a migration approach for an NT4 domain to Active Directory in a mixed mode domain. DLs are used in Exchange 5.5 public folders as security objects. Technically, when the ADC replicated the Exchange 5.5 DLs to AD as Universal Distribution Groups, once a user accesses a public folder where a DL is used as a security object, that Universal Distribution Group should automatically get converted to a Universal Security Group. However, since our domain is in mixed mode, this conversion failed. The result should have been a loss of all permissions on the public folder in question, leaving only the owner with any permissions to the folder. What I found in this test is that everything still worked, users of both Exchange 5.5 and Exchange 2003 were able to use the public folders without a problem. I called Microsoft support on this issue and they were not able to explain why this worked, as they agreed with their KB articles that it should have caused permissions problems on all public folders using DLs on the client permissions of public folders.]]>
In early March an initial attempt to install Exchange 2003 into our existing Exchange 5.5 site was attempted. However, an unanticipated disaster followed this attempt that resulted in the contents of the Exchange 5.5 Directory on all Exchange 5.5 servers to be removed. This resulted in loss of functionality of all DLs and loss of custom recipients.
The Exchange Directory was eventually restored from backup on our primary Exchange 5.5 Server, however we quickly noticed that the restored Directory was not replicating back to the other Exchange 5.5 servers. I was able to find an old application called authrest.exe that can be used to increase the USN (unique identifier) of the Exchange Directory on a server to force it to be the authoritative Directory Server. This will force Directory informationto be replicated to the other Exchange servers. This process was run on our primary Exchange 5.5 Server which did allow the Directory information to replicate back to the other Exchange servers. Shortly after this we noticed that changes made to Exchange objects on any other server but our primary Exchange 5.5 Server would not replicate back to our primary Exchange 5.5 Server. This was because Bobafetts USN was increased beyond the USN of the other servers, and was so far ahead and the changes on the other servers did not exceed our primary Exchange 5.5 Server USN and therefore would not replicate to our primary Exchange 5.5 Server. Changes made on our primary Exchange 5.5 Server would replicate to the other servers without a problem. I did however notice several side effects of restoring the Exchange Directory on our primary Exchange 5.5 Server. First, Backup Exec stopped working. I started getting Access deniederrors on the backup jobs. Permissions were checked for the backup exec account used to run the backup jobs, and no problems with the permissions were found. I tried using other Exchange admin accounts but I still received Access Denied errors. I then tried creating a new user account, and followed the Microsoft guidelines for assigning a backup account permissions to an Exchange organization. This also did not work.
As a last resort, I tried using NTBACKUP and I found that NTBACKUP was able to access Exchange and proceed to backup the exchange information on our primary Exchange 5.5 Server. I have been running manual backups on our primary Exchange 5.5 Server since then. Apparently something in the restore process of the Exchange Directory on our primary Exchange 5.5 Server has caused some type of problem with Backup Exec. We are running the latest version that is compatible with NT4, so upgrading is not an option.
On Friday, March 31st, another sysadmin and I began running authrest on all exchange 5.5 servers in hopes that authrest would set the same USN on all Exchange 5.5 servers to the same level. This would effectively force all Exchange servers Directories into synch. This theory was based on the emergency recovery documentation that was used during the initial problems in early March.
We ran authrest with an increase value of 101000 on each Exchange server. After the reboot, I tested to see if the replication issues were fixed. I modified a DL by removing my user account from the DL. I then checked that the changes replicated to all other exchange servers. I found initially that it did, so it appeared that the process fixed the replication issues.
Upon further testing and someverification it was verified that the Directory Synch issues are indeed not corrected. Changes made on other Exchange servers are still not replicating to our primary Exchange 5.5 Server. I began to do more in depth research on the authrest application andtried to find more operational details. What I found is that Authrest only increases the existing USN on the Exchange Directory and its objects, it does not reset the USN to the specified value you use when running the application. What this means is that we only accomplished increasing the USN on the Exchange 5.5 servers by 101,000. They are still not in synch. our primary Exchange 5.5 Server replicated the same directory information back to all other Exchange 5.5 servers; authrest did not bring them all to the same level as was previously thought.
Currently our primary Exchange 5.5 Server is still acting as the authoritative Directory server in our Exchange organization. Its USN is the highest among all of our Exchange 5.5 servers. Based on this updated information, here are the options that we have in dealing with this situation…
1. Continue to use our primary Exchange 5.5 Server as the master Exchange 5.5 Directory server. All changes to all exchange objects should be made on this server. This includes DL membership modifications and new user mailbox creation. (Mailboxes for other servers can still be created on our primary Exchange 5.5 Server, under the advanced tab of the new mailbox wizard, simply select the home server you want the mailbox created on).
2. Exchange logs an event to the application log during replication of the Exchange 5.5 Directory. We could identify each servers USN level (or number), then based on these results, use authrest to increase each servers USN to a specific number with the goal of individually increasing each other Exchange 5.5 server to thesame number. So authrest would be run with different increment numbers based on that servers existing USN number. Using this method we can correctly use authrest to re-synch the Exchange 5.5 Directory.
I recommend option 1. I dont see any real need to cause further exchange downtime to fix an issue that wont cause us any problems in the Exchange 2003 migration. I can live with running manual backups on our primary Exchange 5.5 Server until its replaced. Backup and restore on our primary Exchange 5.5 Server using NTBACKUP was verified to be working.
The emergency documentation used to restore the Exchange 5.5 Directory in early March did not include the functional information on authrest that is now known. The attempt that I made to re-synch the Directory was based on the information I had of authrest from the emergency restore that was performed earlier. It took several hours of research to find more information on this tool as it relates to Exchange 5.5. Most of the available information is in reference to an updated version that runs on Exchange 2003. Only passing references to this tool were located. Since support for Exchange 5.5 has ended, its been increasingly difficult to find information on Exchange 5.5.
Continuing to use our primary Exchange 5.5 Server as the authoritative Directory server will not adversely affect the Migration to Exchange 2003. If all changes to Exchange 5.5 objects are made onour primary Exchange 5.5 Server, it will always have the updated and correct Directory information. We can then proceed as planned and use our primary Exchange 5.5 Server in the ADC connector as the source of Exchange 5.5 Directory information to be imported into AD and Exchange 2003. By using option 2 above, any exchange 5.5 servers could be used for the ADC connector, but only 1 server is needed. ???
Once Exchange 2003 is successfully implemented for the first time, it will be much easier to add more servers later on. We can then migrate users and other Exchange 5.5 objects to Exchange 2003. From that point on, most of these issues we have been experiencing lately should be a thing of the past. The new infrastructure will prove to be much more resilient and easier to work with.
NOTE: Changes made on our primary Exchange 5.5 Server may not be immediately visible on the other Exchange 5.5 servers. Replication should take place within 5-15 minutes. A manual directory refresh can be done through the Exchange Administrator by highlighting the exchange server you want to update, double click on the Directory Service, and then click Update Now. Choose the option to update only new or changed items. Then click ok. This will force the remote server to update its Directory.
Thanks for reading! Hope this helps someone!]]>
Well it looks like I’m going to be in New York again in a few weeks. Hopefully it will only be a few days and not a whole week. I have one thing that I really look forward to while in New York, and thats meals. I can feast on fruits and breads and all the things I love but don’t get to eat normally due to cost and freshness issues! There is a really great deli in the building the New York office is in. I go there for breakfast, lunch and dinner. Although Belmora pizza is my favorite pizza joint that I have to stop in for a bite at least once while I’m in the area. I’m not certain of the dates yet, but I think I’m definately going. I’ve been to New York a lot lately, and unfortunately, I don’t see it slowing down anytime soon.