Blog Archives

Current problem – seeking solution (backup related)

Let me expalin a little about a problem I am trying to solve. This relates to backups using Backup Exec 10 and 11. I am hoping to get some feedback and ideas on a solution, your comments and ideas are welcome. Please feel free to post to the comments on this article.

First, we have servers all over the place and back them up locally when possible. So each major location has a consolidated backup server with over 1TB of hard drive space and utilizes a backup to disk strategy for the daily backups, and then we have a separate archive to tape job which puts the B2D files onto tape. We keep 3 days worth of backup to disk files on each backup server, using media set controls with a overwrite protection period of 2 days so we only ever have 3 days of backups in each media set.

Jobs are configured with backup to disk folders to keep the media in. Now the issue comes as our storage grows, we are backing up more and more data. Keeping 3 days worth of data on the server gets harder and harder. Not to mention we also are currently backing up 3 days worth of backup to disk files to tape each day, which eats through more than one ultrium tape per day.

The goal I am working towards is being able to keep 3 days worth of backup to disk files on the backup server (idea is to have quick access to restore media 3 days back in case something important needs to be restored) but we only want to backup 1 day at a time to tape, so we don’t waste tape storage and have some crazy confusing tape rotation schedule. We use an autoloader with 24 tape capacity, and are having difficulty in maintaining a good predictable rotation.

So does anyone reading this have any suggestions for being able to keep 3 days worth of backup to disk files on the server, but support backing up only 1 days worth of files at a time to tape? Currently the backup to disk folders are created and named after each server.

I also want to find a way to prevent having to use third party software to keep the backup to disk folders cleaned out. Sometimes the B2D files don’t get re-used for a while, they are protected, but don’t get re-used and just sit there taking up space and ultimately eat up valuable hard drive space for nothing. I had been using some external software to clean out files older than 3 days from these folders.

All suggestions are welcome.

Now running WordPress 2.2

As of late last night I am now running WordPress 2.2. I was a little nervous about the upgrade, and it was a little complicated, but it went ok. I did run into a few interesting situations…

First, I use the plugin “Category Access” which restricts some categories from being publically avialable. These I want to keep privately for my own viewing. So when I went to do the upgrade, I saw the docs wanted you to disable all plugins. Well if I did that, my private categories would be exposed to the world and I didn’t want that. So I left category access enabled.

Second, my FTP client was not set to overwrite properly, so my first run through was not successful. Thank goodness I made backups before I started. Second time was the charm for me.

As far as my plugins go, everything still works except for two minor issues. I wanted to try sidebar widgets on my blog, but after upgrading to 2.2, I found that sidebar widgets won’t activate properly, and generates a fatal error. The plugin cannot be activated. Also, my category access plugin now locks the subject of some messages saying I don’t have the proper permissions to view the posts, however I can click on the posts and view them normally. So there is a small bug in category access, but overall I’m pleased with the upgrade.

Computer problems

Sometimes even us computer professionals make really dumb mistakes on our computers. I was downloading some software last night and messing around a bit, when I opened something I knew I shouldn’t have. I had some zip files on my desktop that I had downloaded and when I tried to open them (without first scanning them for viruses) they just said the archive was damaged. But I could extract them with winrar. So I figured I just did something bad! I was adding and removing software from the PC, when I had to reboot. After the reboot, Windows would load to a blue screen waiting to open the logon box, but instead I just got STOP errors and Winlogon would crash and do a memory dump. So I stayed up till about 11:30 working on it, trying to get it working before Liz got home so she would have her PC today. I was never able to get it working and even tried some live linux CDs just to get her something to use. No luck! So I am going to try a different tool tonight to see if I can get to the system restore feature of Windows XP (since not even Safe Mode will work). If that works it makes it easier for me to backup and then re-load everything, but either way, it looks I’ll spend my weekend re-loading our PC again! I seem to be doing that more and more frequently lately. I wish I had a nice powerful server I could run Virtual server 2005 on, then I could just setup a virtual PC for all my downloading and testing, then it wouldn’t screw up my PC! I think I may also start making routine backups with ghost or something like that once I get it back up and running. I get the feeling that I need to do some hardware swapping too, since the machine seems so slow. Its got good stuff in it, but its just not as fast as it used to be. I could probably use a new hard drive, memory and motherboard. We use our computers a lot, and they run almost all the time, so things do tend to wear out!

Active Directory Troubles

UPDATE: 11-21-06

I was able to resolve my issues last night. The transition from a single DC to a multi DC and back again went very smoothly. It also gave me a chance to redo the computer names a bit more geared towards Battlestar Galactica characters. Sisko has now been renamed to Adama, and is once again the only DC on my home network. I plan to take another server currently in pieces, and put it back into use as a backup server, it will literally be my backup server. This machine will handle physical backups of the other servers and also be an additional DC for my AD network. I hope to have everything finished by the end of the long weekend!]]>

TECH: Authrest on Exchange 5.5

Original Problem???

In early March an initial attempt to install Exchange 2003 into our existing Exchange 5.5 site was attempted. However, an unanticipated disaster followed this attempt that resulted in the contents of the Exchange 5.5 Directory on all Exchange 5.5 servers to be removed. This resulted in loss of functionality of all DLs and loss of custom recipients.

The Exchange Directory was eventually restored from backup on our primary Exchange 5.5 Server, however we quickly noticed that the restored Directory was not replicating back to the other Exchange 5.5 servers. I was able to find an old application called authrest.exe that can be used to increase the USN (unique identifier) of the Exchange Directory on a server to force it to be the authoritative Directory Server. This will force Directory informationto be replicated to the other Exchange servers. This process was run on our primary Exchange 5.5 Server which did allow the Directory information to replicate back to the other Exchange servers. Shortly after this we noticed that changes made to Exchange objects on any other server but our primary Exchange 5.5 Server would not replicate back to our primary Exchange 5.5 Server. This was because Bobafetts USN was increased beyond the USN of the other servers, and was so far ahead and the changes on the other servers did not exceed our primary Exchange 5.5 Server USN and therefore would not replicate to our primary Exchange 5.5 Server. Changes made on our primary Exchange 5.5 Server would replicate to the other servers without a problem. I did however notice several side effects of restoring the Exchange Directory on our primary Exchange 5.5 Server. First, Backup Exec stopped working. I started getting Access deniederrors on the backup jobs. Permissions were checked for the backup exec account used to run the backup jobs, and no problems with the permissions were found. I tried using other Exchange admin accounts but I still received Access Denied errors. I then tried creating a new user account, and followed the Microsoft guidelines for assigning a backup account permissions to an Exchange organization. This also did not work.

As a last resort, I tried using NTBACKUP and I found that NTBACKUP was able to access Exchange and proceed to backup the exchange information on our primary Exchange 5.5 Server. I have been running manual backups on our primary Exchange 5.5 Server since then. Apparently something in the restore process of the Exchange Directory on our primary Exchange 5.5 Server has caused some type of problem with Backup Exec. We are running the latest version that is compatible with NT4, so upgrading is not an option.

Synchronization Information???

On Friday, March 31st, another sysadmin and I began running authrest on all exchange 5.5 servers in hopes that authrest would set the same USN on all Exchange 5.5 servers to the same level. This would effectively force all Exchange servers Directories into synch. This theory was based on the emergency recovery documentation that was used during the initial problems in early March.

We ran authrest with an increase value of 101000 on each Exchange server. After the reboot, I tested to see if the replication issues were fixed. I modified a DL by removing my user account from the DL. I then checked that the changes replicated to all other exchange servers. I found initially that it did, so it appeared that the process fixed the replication issues.

Upon further testing and someverification it was verified that the Directory Synch issues are indeed not corrected. Changes made on other Exchange servers are still not replicating to our primary Exchange 5.5 Server. I began to do more in depth research on the authrest application andtried to find more operational details. What I found is that Authrest only increases the existing USN on the Exchange Directory and its objects, it does not reset the USN to the specified value you use when running the application. What this means is that we only accomplished increasing the USN on the Exchange 5.5 servers by 101,000. They are still not in synch. our primary Exchange 5.5 Server replicated the same directory information back to all other Exchange 5.5 servers; authrest did not bring them all to the same level as was previously thought.

Options

Currently our primary Exchange 5.5 Server is still acting as the authoritative Directory server in our Exchange organization. Its USN is the highest among all of our Exchange 5.5 servers. Based on this updated information, here are the options that we have in dealing with this situation…

1. Continue to use our primary Exchange 5.5 Server as the master Exchange 5.5 Directory server. All changes to all exchange objects should be made on this server. This includes DL membership modifications and new user mailbox creation. (Mailboxes for other servers can still be created on our primary Exchange 5.5 Server, under the advanced tab of the new mailbox wizard, simply select the home server you want the mailbox created on).
2. Exchange logs an event to the application log during replication of the Exchange 5.5 Directory. We could identify each servers USN level (or number), then based on these results, use authrest to increase each servers USN to a specific number with the goal of individually increasing each other Exchange 5.5 server to thesame number. So authrest would be run with different increment numbers based on that servers existing USN number. Using this method we can correctly use authrest to re-synch the Exchange 5.5 Directory.

I recommend option 1. I dont see any real need to cause further exchange downtime to fix an issue that wont cause us any problems in the Exchange 2003 migration. I can live with running manual backups on our primary Exchange 5.5 Server until its replaced. Backup and restore on our primary Exchange 5.5 Server using NTBACKUP was verified to be working.

Summary Information???

The emergency documentation used to restore the Exchange 5.5 Directory in early March did not include the functional information on authrest that is now known. The attempt that I made to re-synch the Directory was based on the information I had of authrest from the emergency restore that was performed earlier. It took several hours of research to find more information on this tool as it relates to Exchange 5.5. Most of the available information is in reference to an updated version that runs on Exchange 2003. Only passing references to this tool were located. Since support for Exchange 5.5 has ended, its been increasingly difficult to find information on Exchange 5.5.

Continuing to use our primary Exchange 5.5 Server as the authoritative Directory server will not adversely affect the Migration to Exchange 2003. If all changes to Exchange 5.5 objects are made onour primary Exchange 5.5 Server, it will always have the updated and correct Directory information. We can then proceed as planned and use our primary Exchange 5.5 Server in the ADC connector as the source of Exchange 5.5 Directory information to be imported into AD and Exchange 2003. By using option 2 above, any exchange 5.5 servers could be used for the ADC connector, but only 1 server is needed. ???

Once Exchange 2003 is successfully implemented for the first time, it will be much easier to add more servers later on. We can then migrate users and other Exchange 5.5 objects to Exchange 2003. From that point on, most of these issues we have been experiencing lately should be a thing of the past. The new infrastructure will prove to be much more resilient and easier to work with.

NOTE: Changes made on our primary Exchange 5.5 Server may not be immediately visible on the other Exchange 5.5 servers. Replication should take place within 5-15 minutes. A manual directory refresh can be done through the Exchange Administrator by highlighting the exchange server you want to update, double click on the Directory Service, and then click Update Now. Choose the option to update only new or changed items. Then click ok. This will force the remote server to update its Directory.

Thanks for reading! Hope this helps someone!]]>

Disaster in New York

I was in the office that night (Tuesday) until 3am (Wednesday) trying to fix this problem. I was able to restore the Exchange 5.5 Directory from a backup, but this took hours to find a good backup, restore the data, then fix Exchange so it would replicate to the other Exchange 5.5 Servers. Unfortunately, we still have problems even after the restore. We experienced issues with our permissions on our public folders. This is a major pain because no one had a list of who had permissions to what, so I’m shooting in the dark trying to fix it. I can only fix these problems when users complain that they don’t have access to what they need.

This was not a good week. I am so glad I’m sitting in the airport in New Jersey right now waiting for my flight to leave for Tampa. It will be so nice to get home! Maybe its time to think about a vacation! Fortunately, everything is operationally back up and working now, there are just a few minor things like end user permissions to work out as they pop up. Oh and one other issue. Now my backups on my exchange server that I restored the Directory on are failing. I get access denied errors, even though the account I’m using in backup exec has all the permissions it needs to complete an Exchange backup. So I’m going to be working on that ASAP.]]>

Backups backups backups

After about 9 years of being in IT, I’ve learned how important backups are. I’ve experienced data loss due to not having a good backup several times. It really is worth it to have a good backup plan. Whether at work or at home. At home, I’ve got two 2003 servers, two XP desktops and I always make sure I at least make a ghost image of each one, with just the bare OS and again with all the software I need loaded. For example, my home PC, last night I was having troubles due to some beta software I installed (foolishly). I had planned to use my other XP desktop for software testing, but never seem to do that. The junk always goes on my primary PC, after all its faster and resist crashing longer than the slower desktop. So anyway, long story short, I decide to restore a ghost image of my Home PC with all the software loaded on it. Since I have a domain, my user accounts are still OK, I keep all my data on separate drives and network locations, so I won’t loose anything. But there always seems to be something you forget. Items like desktop downloads, favorites, RSS feeds, etc. now I couldn’t get windows explorer to work long enough to copy anything, so I got out my trusty Knoppix 3.9 bootable CD, inserted my thumb drive, and off I went, I was able to use Linux to get my data off my drive, before I reloaded the ghost image. I finally around 11pm, got all the stuff copied over and my ghost image restored. All I had to do was setup our desktop profiles again, like setting up outlook, and redirecting the my documents folder. Then I was done. So I’m all back up and running, thanks to having a good backup strategy. It only took 4 minutes to restore the ghost image, if I had to reinstall windows and all my apps manually, it would have taken hours! I use NTBACKUP to backup my two servers. The other desktop I don’t care about and will likely give away soon anyway.

Once I lost 2 years of data on a Novell Netware server I had at home. I decided I’d break a mirrored volume in Netware, forgetting I had stuff on it. Once it was broken, my data was gone. Boo hoo hoo. I was recently using a Dell PowerVault NAS 705N for my storage needs, but its a little slow, it was RAID5 which was good, but I decided to sell it, so now its listed on eBay. I hope I can sell it and get another 400GB hard drive. I have about 1TB of storage space in my PC right now. I need it too, I’ve got 200GB of normal data, like CD images, software, downloads, documents, etc. Then I have lost of multimedia, like music, movies, home video recordings and such that take up a lot of space. So I need to get one more drive and that should do it for now. My only concern now is that if any one of these drives fail, I loose all my data, and I don’t have a location on my network with storage large enough to store all that data if I were to back it up. I thought about getting a tape drive, but don’t want to spend the money (that I don’t have) on a tape backup solution which would likely not be large enough to hold all my stuff on a single tape anyway.
]]>