Blog Archives

Moved to new domain

For those of you who read my blog or subscribe to my RSS feed, you should know that I have moved to a new domain. All my old links/posts/pages, etc should all still work, but you may see some strange issues now and then until you update your bookmarks and or RSS feed link. My new domain name is joechurch.com and my blog link is now located at http://www.joechurch.com/blog

I will be putting up a new site at the root of joechurch.com sometime soon, but for now the only thing that works is my blog link.

It was quite a challenge to move domains with my blog, as I had lots of plugin issues and link problems. I ended up starting from scratch and re-installing wordpress and all my plugins manually. I used a wordpress export to move everything over which worked great except that I lost of blogroll links. I was able to use the wayback machine at http://www.archive.org to get the links I used to have and the names I gave them. So for now, the blog is fully moved over and working great.

Goodlink 5 deployment issue

I have been working on deploying multiple Goodlink 5 messaging servers recently, and came across a problem with the user migration feature in Goodlink 5.  Normally if you have more than one Good messaging server, you have the option to “change good messaging server” when you right-click on a user account.  So in my deployment when I didn’t see this option enabled, I had to call Goodlink support.  What I found out made sense, but is definately irritating. 

I called Goodlink support before deploying Goodlink version 5 to ask about deployment options and get some advice on my project.  I asked about the user migration feature and using multiple GoodAdmin mailboxes, and was told that it should work fine.  Now here is where the issue comes in.  In my deployment scenario, I wanted to deploy multiple individual Goodlink 5 servers in various locations/offices around the world.  The limitation here is network bandwidth and latency, due to the fact that the offices getting these Goodlink 5 servers are spread across the globe on various speed networks, and some of the offices in Asia for example, have very high network latency and bandwidth is limited.  To help with this situation, I asked if it would be advisable to use multiple GoodAdmin mailboxes, one for each new Goodlink 5 server, that way, each location that got a Goodlink 5 server and had a local Exchange server, would get its own GoodAdmin mailbox homed on the local Exchange server.  This would increase performance, reduce the time it takes to re-connect all the mailboxes in the event of a reboot, and provided a better solution for my deployment project. 

When you use a single consolidated GoodAdmin mailbox, and you have more than one Goodlink server which is connected to the network on a low bandwidth link or one that has high latency, it is a good idea to have separate GoodAdmin mailboxes, to keep network traffic local to the office in question.  Its faster, more reliable, and if the central Exchange server that houses the consolidated GoodAdmin mailbox were to go down, all your Good Messaging servers would be rendered useless until the Exchange server is back online and the GoodAdmin mailbox is available again. 

So my scenario of using more than a single consolidated GoodAdmin mailbox was a good idea, still works well and does help with performance and all the other reasons I mentioned above.  The drawback is that in this configuration, from within the Good Messaging console, you only see the server you are connected to under the “Good messaging servers” folder.  This also means that the “change good messaging server” option is grayed out and unavailable.  The fact is that the GoodAdmin mailbox is what keeps track of how many servers you have, but they all have to be configured to use the same GoodAdmin mailbox in order for the server to “know” about the other Good Messaging servers you have in your network.  This basically means that the user migration option is not going to work for us. 

This is not a problem, its just an inconvenience.  And it would have been nice to have known this initially when I talked with Good support eariler.  Lately no matter what company I deal with, I usually end up getting conflicting information about their products and features based on who you talk to.  The first time I call and get info on something, I hear one thing, then when I am working on the deployment and have a question or problem and call back, I hear something completely different.  This is irritating and can cause some major problems for large projects. 

When NT4 servers can't find the PDC and all else fails

Recently I ran into a major problem with my Active Directory and NT4 setup. I maintain a network made up of 2 NT4 BDCs and about 10 Active Directory Domain Controllers. The domain is in 2003 interim mode and we also run Exchange 5.5 on 4 other NT4 member servers. Last week we renamed a few domain controllers and assigned new IP addresses (on the 2003 side). As a precaution I kept the old IP address on one of the major domain controllers until I could get time to manually modify all the legacy servers lmhosts files. We also shutdown the domain controller that was used to get us to Active Directory, basically I took a dell desktop that would run NT, and made it a BDC. Then I promoted it to a PDC and upgraded the OS to Server 2003 and installed Active Directory. Well its time to de-commission that box and we shut it down last week as well. On Monday, I created a new user account and immediately got reports of strange problems, the user was getting prompted for logon credentials in Outlook and could not stay online.

I looked around and couldn’t find anything wrong and wasn’t too concerned at this point. Later I realized that my NT4 BDCs were not able to find a PDC any longer. I assumed it was because we shutdown the upgraded domain controller and so we powered it back on hoping it would help. This did not fix the problem so I began working on the issue by online research and posting questions in newsgroups. Finally I found a guy on Experts-exchange that was very helpful and worked with me on EE for hours before we figured out the issue. By troubleshooting and much testing we found that NetBIOS lookups to the PDC emulator (running 2003) were failing. From 2003 we could map drives, browse to NT4 without a problem, only from NT4 to 2003 was there an issue. Lastly we found that its bad to have a multi-homed domain controller, especially the one we were using for the PDC Emulator. I removed the second (old) IP address from the server and everything started to work just fine. I could get into the user manager in NT4 and updates started to be processed without a problem.

So it turns out the main cause of the issue was not the renames, or IP change, or even shutting down an old DC. It was simply that we had more than 1 IP Address on our PDC Emulator server. Removing that fixed the issue. I think we can now power down the upgraded DC again and proceed further with the migration. Too bad it took so long to figure this out, but at least it is working normally now.

Surprise Weekend

This is going to be a rather long story. Its a tale of surprise and fun.

It all started on Thursday evening while I was at work. I was working on a server migration project involving Goodlink, when things took a turn for the worse. I was on the phone with Goodlink support almost all day. I was having a terrible time getting things to work properly, and ended up having to stay super late on Thursday, I didn’t leave the office until 11pm. I get in to work around 7:45am or so every weekday, so I was at work from 7:45am until 11m, which makes for a very long day! I finally got things stabalized with the help of Goodlink support and started on my way home. Once I got home, around 11:30pm, I walked in the door, hungry (since I hadn’t eaten anything) and tired (obviously), and so I sat down next to Liz who was sitting on the couch. I leaned over to give her a hello kiss, when I saw something out of the corner of my eye that startled me. It was my borhter Jeremy who’s hand was reaching up towards me in an attempt to surprise me! I jumped up off the couch in surprise and realized it was him. They were totally playing me and had even more surprised in store for me.

We talked for a bit and got a snack and a drink and finally went to bed around 12:30am. The next morning we got up around 8 or so, and I was still in bed when there was a ring at the doorbell. I had a suspicion as to who it might be, but Liz asked me to get the door, so I did and found my mom standing out on the porch. Turned out they had both flown in that Thursday morning and spent the day with Liz and the kids. I had to work late so it totally screwed up their surprise plans and they had to completely change around what they were going to do. So we got to spend the whole weekend (I took Friday off too) with my mom and Jeremy. Click more or the post title to read more about the weekend…
Read the rest of this entry

TECH: Exchange Migration and permissions testing

Introduction

This posting will review the approach taken in a test environment to accomplish a simulation migration of our existing NT4 domain to server 2003 with Exchange 2003. All work was done offline on a private network IP scheme on an isolated switch to prevent communication problems with the production network. Access was limited to a hardwired laptop and a test desktop. Testing was performedwith a user/mailbox on Exchange 5.5, and another user/mailbox on Exchange 2003. Migration Process

The following are the steps in order that were used to perform a test migration of the NT4 domain. Directory exports from Exchange 5.5 were used to re-create the Exchange 5.5 Directory in a test environment.

Server Setup

? Initial NT4 PDC was create for the domain utilizing VMWARE and was updated to SP6A. Names utilized were duplicates of production network.
? Two additional NT4 BDCs were created;one to simulate a BDC, the other was create to perform the upgrade to Server 2003 and AD. Both were installed as BDCs and were updated to SP6A.
? Three NT4 member servers were created for use with Exchange 5.5. All were installed as member servers, and updated to SP6A. Each server was named the same as in production.
? Exchange 5.5 was installed on a member NT4 server, creating a duplicate of our existing Exchange 5.5 system. Org and site names were duplicated from Production.
? Exports of the Exchange Directory were imported and were also used to create the NT4 user accounts/mailboxes used in production. This populated the NT4 user database with usernames and blank passwords. New Mailboxes were created for users on their respective servers
? A test workstation was setup on the test NT4 domain. An exchange outlook profile was created for the exchange 5.5 user.
? Tests were performed to verify that DLs and inter-org mail flow was working.
? The last NT4 BDC that was created initially was promoted to a PDC (automatically making the original PDC a BDC).
? The OS on the new PDC was upgraded to Server 2003.
? The Active Directory Installation Wizard was run to upgrade our domain to AD.
? The AD wizard installed DNS locally on the PDC, but HOSTS files were still maintained on all NT4 servers, just as in production.
? A desktop was hardwired to the isolated switch and setup as an AD-client using a user account in AD. This was to verify that AD had successfully upgraded our NT4 domain to AD.
? A new server was installed also using VMWARE using server 2003 operating system. Server was installed as a member server of the AD domain. This was for use with the migration to Exchange 2003. SP1 was installed on the server and all updates and patches were applied.
? A temporary OU was created in AD to house the objects from the ADC replication process.
? Exchange 2003 was installed on this new 2003 server, along with the ADC and SRS. This new Exchange 2003 server was joined to our existing Exchange 5.5 site.
? The ADC replicated all Exchange 5.5 objects such as DLs, custom recipients, etc, into AD. The DLs were replicated to AD as universal distribution groups.
? Testing was performed between Exchange 5.5 users and Exchange 2003 users to verify mail flow and DL functionality. All tests completed successfully. No loss of public folder permissions was experienced. Although error 9552 in the Exchange 2003 server event log was experienced. But no loss of permissions was observed.

The above information was a test of a migration approach for an NT4 domain to Active Directory in a mixed mode domain. DLs are used in Exchange 5.5 public folders as security objects. Technically, when the ADC replicated the Exchange 5.5 DLs to AD as Universal Distribution Groups, once a user accesses a public folder where a DL is used as a security object, that Universal Distribution Group should automatically get converted to a Universal Security Group. However, since our domain is in mixed mode, this conversion failed. The result should have been a loss of all permissions on the public folder in question, leaving only the owner with any permissions to the folder. What I found in this test is that everything still worked, users of both Exchange 5.5 and Exchange 2003 were able to use the public folders without a problem. I called Microsoft support on this issue and they were not able to explain why this worked, as they agreed with their KB articles that it should have caused permissions problems on all public folders using DLs on the client permissions of public folders.]]>

TECH: Authrest on Exchange 5.5

Original Problem???

In early March an initial attempt to install Exchange 2003 into our existing Exchange 5.5 site was attempted. However, an unanticipated disaster followed this attempt that resulted in the contents of the Exchange 5.5 Directory on all Exchange 5.5 servers to be removed. This resulted in loss of functionality of all DLs and loss of custom recipients.

The Exchange Directory was eventually restored from backup on our primary Exchange 5.5 Server, however we quickly noticed that the restored Directory was not replicating back to the other Exchange 5.5 servers. I was able to find an old application called authrest.exe that can be used to increase the USN (unique identifier) of the Exchange Directory on a server to force it to be the authoritative Directory Server. This will force Directory informationto be replicated to the other Exchange servers. This process was run on our primary Exchange 5.5 Server which did allow the Directory information to replicate back to the other Exchange servers. Shortly after this we noticed that changes made to Exchange objects on any other server but our primary Exchange 5.5 Server would not replicate back to our primary Exchange 5.5 Server. This was because Bobafetts USN was increased beyond the USN of the other servers, and was so far ahead and the changes on the other servers did not exceed our primary Exchange 5.5 Server USN and therefore would not replicate to our primary Exchange 5.5 Server. Changes made on our primary Exchange 5.5 Server would replicate to the other servers without a problem. I did however notice several side effects of restoring the Exchange Directory on our primary Exchange 5.5 Server. First, Backup Exec stopped working. I started getting Access deniederrors on the backup jobs. Permissions were checked for the backup exec account used to run the backup jobs, and no problems with the permissions were found. I tried using other Exchange admin accounts but I still received Access Denied errors. I then tried creating a new user account, and followed the Microsoft guidelines for assigning a backup account permissions to an Exchange organization. This also did not work.

As a last resort, I tried using NTBACKUP and I found that NTBACKUP was able to access Exchange and proceed to backup the exchange information on our primary Exchange 5.5 Server. I have been running manual backups on our primary Exchange 5.5 Server since then. Apparently something in the restore process of the Exchange Directory on our primary Exchange 5.5 Server has caused some type of problem with Backup Exec. We are running the latest version that is compatible with NT4, so upgrading is not an option.

Synchronization Information???

On Friday, March 31st, another sysadmin and I began running authrest on all exchange 5.5 servers in hopes that authrest would set the same USN on all Exchange 5.5 servers to the same level. This would effectively force all Exchange servers Directories into synch. This theory was based on the emergency recovery documentation that was used during the initial problems in early March.

We ran authrest with an increase value of 101000 on each Exchange server. After the reboot, I tested to see if the replication issues were fixed. I modified a DL by removing my user account from the DL. I then checked that the changes replicated to all other exchange servers. I found initially that it did, so it appeared that the process fixed the replication issues.

Upon further testing and someverification it was verified that the Directory Synch issues are indeed not corrected. Changes made on other Exchange servers are still not replicating to our primary Exchange 5.5 Server. I began to do more in depth research on the authrest application andtried to find more operational details. What I found is that Authrest only increases the existing USN on the Exchange Directory and its objects, it does not reset the USN to the specified value you use when running the application. What this means is that we only accomplished increasing the USN on the Exchange 5.5 servers by 101,000. They are still not in synch. our primary Exchange 5.5 Server replicated the same directory information back to all other Exchange 5.5 servers; authrest did not bring them all to the same level as was previously thought.

Options

Currently our primary Exchange 5.5 Server is still acting as the authoritative Directory server in our Exchange organization. Its USN is the highest among all of our Exchange 5.5 servers. Based on this updated information, here are the options that we have in dealing with this situation…

1. Continue to use our primary Exchange 5.5 Server as the master Exchange 5.5 Directory server. All changes to all exchange objects should be made on this server. This includes DL membership modifications and new user mailbox creation. (Mailboxes for other servers can still be created on our primary Exchange 5.5 Server, under the advanced tab of the new mailbox wizard, simply select the home server you want the mailbox created on).
2. Exchange logs an event to the application log during replication of the Exchange 5.5 Directory. We could identify each servers USN level (or number), then based on these results, use authrest to increase each servers USN to a specific number with the goal of individually increasing each other Exchange 5.5 server to thesame number. So authrest would be run with different increment numbers based on that servers existing USN number. Using this method we can correctly use authrest to re-synch the Exchange 5.5 Directory.

I recommend option 1. I dont see any real need to cause further exchange downtime to fix an issue that wont cause us any problems in the Exchange 2003 migration. I can live with running manual backups on our primary Exchange 5.5 Server until its replaced. Backup and restore on our primary Exchange 5.5 Server using NTBACKUP was verified to be working.

Summary Information???

The emergency documentation used to restore the Exchange 5.5 Directory in early March did not include the functional information on authrest that is now known. The attempt that I made to re-synch the Directory was based on the information I had of authrest from the emergency restore that was performed earlier. It took several hours of research to find more information on this tool as it relates to Exchange 5.5. Most of the available information is in reference to an updated version that runs on Exchange 2003. Only passing references to this tool were located. Since support for Exchange 5.5 has ended, its been increasingly difficult to find information on Exchange 5.5.

Continuing to use our primary Exchange 5.5 Server as the authoritative Directory server will not adversely affect the Migration to Exchange 2003. If all changes to Exchange 5.5 objects are made onour primary Exchange 5.5 Server, it will always have the updated and correct Directory information. We can then proceed as planned and use our primary Exchange 5.5 Server in the ADC connector as the source of Exchange 5.5 Directory information to be imported into AD and Exchange 2003. By using option 2 above, any exchange 5.5 servers could be used for the ADC connector, but only 1 server is needed. ???

Once Exchange 2003 is successfully implemented for the first time, it will be much easier to add more servers later on. We can then migrate users and other Exchange 5.5 objects to Exchange 2003. From that point on, most of these issues we have been experiencing lately should be a thing of the past. The new infrastructure will prove to be much more resilient and easier to work with.

NOTE: Changes made on our primary Exchange 5.5 Server may not be immediately visible on the other Exchange 5.5 servers. Replication should take place within 5-15 minutes. A manual directory refresh can be done through the Exchange Administrator by highlighting the exchange server you want to update, double click on the Directory Service, and then click Update Now. Choose the option to update only new or changed items. Then click ok. This will force the remote server to update its Directory.

Thanks for reading! Hope this helps someone!]]>

Company Migration Project

1. Little to no end user interruption
2. No loss of email connectivity
3. All work must be done by internal IT staff
4. No third party tools can be used

Existing Network:

1 NT4 PDC in NYK and 1 NT4 BDC in Switzerland. Two Exchange 5.5 servers in NYK, one in Switzerland and one in Hong Kong. There are also various software packages that sit on top of the Exchange infrastructure that must remain intact, such as RightFax (email faxing), Goodlink (mobile email) etc. Exchange is installed on member servers, none of the Exchange servers are installed on a PDC/BDC.

Desired result:

2003 Active Directory Domain with Exchange 2003 Enterprise Edition. Exchange servers will be in two-server clusters. 4 hub offices will house the Exchange clusters.

Migration Path:

In order to accomplish the above design goals and satisfy all the requirements, taking into account the existing network infrastructure, the following design should prove to be successful.

First, we will install a new server into our existing NT4 domain and make it a BDC for the domain. We will then, promote the new BDC to a PDC and allow time for replication. Once replication is complete, we will take the old PDC offline (by unplugging network connection or shutting down). We will then upgrade the new NT4 PDC to Server 2003 and run DCPROMO to install Active Directory. This process should preserve all existing user accounts, machine accounts, groups, permissions, etc. So far we satisfy all requirements.

Next, we upgrade or replace all existing NT4 BDCs. Once all NT4 servers are removed, we can then upgrade our new Active directory domain to Server 2003 Native Mode. As no more NT4 servers will be participating as domain controllers.

NOTE: During this time, all existing Exchange 5.5 servers will be maintained as NT4 member servers of the 2003 domain. Exchange 5.5 will continue to handle mail for us until the upgrade to Exchange 2003.

Exchange 2003 Migration:

For this part, we will install our first Exchange 2003 server (on a 2003 Enterprise Edition server installed as a member server) into our existing Exchange 5.5 site. To the exchange site, we are just adding a new server. The Exchange deployment tools will walk us through installing the Active Directory Connector and all necessary connection agreements. The SRS will also be installed. One Exchange 2003 server per hub office will be installed initially. Once we verify that the ADC is working properly and the “Move mailbox” wizard is available, all Exchange 5.5 user mailboxes will be moved to an Exchange 2003 server. Once all Exchange 5.5 mailboxes, public folders, distribution lists, custom forms, etc, are replicated over to the new Exchange 2003 servers, the existing Exchange 5.5 servers will be shut down to verify connectivity and that no “behind the scenes” issues exist. Once our Exchange organization has been verified to be functioning correctly with no further references to the Exchange 5.5 servers, we can then begin to de-commission them. This will be done by removing all references to Exchange 5.5 as replication partners on all public folders, and other exchange resources. Then the servers can be deleted from the Exchange 5.5 server administrator.

During this migration process, the only end user interruption noticed will be during the move mailbox process, as users will be logged out of their exchange mailboxes during a mailbox move. Mail flow is not affected since we installed Exchange 2003 into our existing Exchange 5.5 site, so the routing group is the same. The only remaining task to complete is something I left out above. Before Exchange 5.5 can be shut down or de-commissioned, the SMTP connector will need to be moved from one of the Exchange 5.5 servers to one of the Exchange 2003 servers. Once this is complete, and mail flow has been verified, then the Exchange 5.5 servers can be removed.

The end result is a quick, efficient migration/upgrade to Server 2003 and Exchange 2003. A final note here is on Clustering. The reason we did not use our clusters to install the first Exchange 2003 server is that there are certain components of Exchange that will not function on a cluster. Such as the SRS and ADC. This is why we will be using a standalone server in all hub offices for the initial move to Exchange 2003. Once we have Exchange 2003 up and running globally, we can then introduce our Exchange 2003 Clusters and then move mailboxes once more to the clusters. Once finished, the SMTP (bridgehead) can be moved to the appropriate cluster and the initial Exchange 2003 standalone servers can be removed.

The initial plan was to do a parallel migration, basically creating a whole new system in parallel to our existing system. This plan has many problems and would not have worked for us. End users would have received all new machine profiles, outlook profiles would have been lost, etc. This would have created too much work for our internal IT staff and caused too much interruption to end user connectivity. Not to mention mail flow and interoperability with Exchange 5.5 and Exchange 2003 is much more complicated when installed in separate Exchange Organizations. Third party tools would almost certainly be needed to maintain the level of co-existence we would have needed.
]]>

Exchange 2003 and Exchange 5.5 Migration and Co-Existence

Q1. Can you install Exchange 2003 into an Exchange 5.5 Site?
A1. Yes you can, this is actually the preferred method of migration since you can use the “move mailbox” in Exchange Administration to move mailboxes between Exchange servers in the same Exchange Organization. You will need the ADC to make this happen.

Q2. Can distribution lists in Exchange 5.5 be migrated to Exchange 2003?
A2. Yes they can, using the ADC the groups can be imported as distribution groups in 2003/AD

Q3. What gets complicated by installing 2003 in a separate Exchange organization, rather than installing Exchange 2003 into the Exchange 5.5 organization?
A3. If Exchange servers are within the same organization then mailboxes can be
moved easily through ‘Exchange Tasks’ in AD USers and Computers. They also
maintain the same alias’ (very important when users start replying to old
emails, they will be bounced if the mailbox and alias changes) and users
outlook profiles will also be automatically redirected to the new server as
it is within the same organization. In separate organizations, none of the above is true. As far as I know
exmerge is required to migrate mailboxes, you’ll have to create completely
new mailboxes on the new server and reconfigure users outlook profiles at the
desktop. Mailbox alias’ will also likely be an issue.
]]>