Surprise Weekend

This is going to be a rather long story. Its a tale of surprise and fun.

It all started on Thursday evening while I was at work. I was working on a server migration project involving Goodlink, when things took a turn for the worse. I was on the phone with Goodlink support almost all day. I was having a terrible time getting things to work properly, and ended up having to stay super late on Thursday, I didn’t leave the office until 11pm. I get in to work around 7:45am or so every weekday, so I was at work from 7:45am until 11m, which makes for a very long day! I finally got things stabalized with the help of Goodlink support and started on my way home. Once I got home, around 11:30pm, I walked in the door, hungry (since I hadn’t eaten anything) and tired (obviously), and so I sat down next to Liz who was sitting on the couch. I leaned over to give her a hello kiss, when I saw something out of the corner of my eye that startled me. It was my borhter Jeremy who’s hand was reaching up towards me in an attempt to surprise me! I jumped up off the couch in surprise and realized it was him. They were totally playing me and had even more surprised in store for me.

We talked for a bit and got a snack and a drink and finally went to bed around 12:30am. The next morning we got up around 8 or so, and I was still in bed when there was a ring at the doorbell. I had a suspicion as to who it might be, but Liz asked me to get the door, so I did and found my mom standing out on the porch. Turned out they had both flown in that Thursday morning and spent the day with Liz and the kids. I had to work late so it totally screwed up their surprise plans and they had to completely change around what they were going to do. So we got to spend the whole weekend (I took Friday off too) with my mom and Jeremy. Click more or the post title to read more about the weekend…

TECH: Authrest on Exchange 5.5

Original Problem???

In early March an initial attempt to install Exchange 2003 into our existing Exchange 5.5 site was attempted. However, an unanticipated disaster followed this attempt that resulted in the contents of the Exchange 5.5 Directory on all Exchange 5.5 servers to be removed. This resulted in loss of functionality of all DLs and loss of custom recipients.

The Exchange Directory was eventually restored from backup on our primary Exchange 5.5 Server, however we quickly noticed that the restored Directory was not replicating back to the other Exchange 5.5 servers. I was able to find an old application called authrest.exe that can be used to increase the USN (unique identifier) of the Exchange Directory on a server to force it to be the authoritative Directory Server. This will force Directory informationto be replicated to the other Exchange servers. This process was run on our primary Exchange 5.5 Server which did allow the Directory information to replicate back to the other Exchange servers. Shortly after this we noticed that changes made to Exchange objects on any other server but our primary Exchange 5.5 Server would not replicate back to our primary Exchange 5.5 Server. This was because Bobafetts USN was increased beyond the USN of the other servers, and was so far ahead and the changes on the other servers did not exceed our primary Exchange 5.5 Server USN and therefore would not replicate to our primary Exchange 5.5 Server. Changes made on our primary Exchange 5.5 Server would replicate to the other servers without a problem. I did however notice several side effects of restoring the Exchange Directory on our primary Exchange 5.5 Server. First, Backup Exec stopped working. I started getting Access deniederrors on the backup jobs. Permissions were checked for the backup exec account used to run the backup jobs, and no problems with the permissions were found. I tried using other Exchange admin accounts but I still received Access Denied errors. I then tried creating a new user account, and followed the Microsoft guidelines for assigning a backup account permissions to an Exchange organization. This also did not work.

As a last resort, I tried using NTBACKUP and I found that NTBACKUP was able to access Exchange and proceed to backup the exchange information on our primary Exchange 5.5 Server. I have been running manual backups on our primary Exchange 5.5 Server since then. Apparently something in the restore process of the Exchange Directory on our primary Exchange 5.5 Server has caused some type of problem with Backup Exec. We are running the latest version that is compatible with NT4, so upgrading is not an option.

Synchronization Information???

On Friday, March 31^st, another sysadmin and I began running authrest on all exchange 5.5 servers in hopes that authrest would set the same USN on all Exchange 5.5 servers to the same level. This would effectively force all Exchange servers Directories into synch. This theory was based on the emergency recovery documentation that was used during the initial problems in early March.

We ran authrest with an increase value of 101000 on each Exchange server. After the reboot, I tested to see if the replication issues were fixed. I modified a DL by removing my user account from the DL. I then checked that the changes replicated to all other exchange servers. I found initially that it did, so it appeared that the process fixed the replication issues.

Upon further testing and someverification it was verified that the Directory Synch issues are indeed not corrected. Changes made on other Exchange servers are still not replicating to our primary Exchange 5.5 Server. I began to do more in depth research on the authrest application andtried to find more operational details. What I found is that Authrest only increases the existing USN on the Exchange Directory and its objects, it does not reset the USN to the specified value you use when running the application. What this means is that we only accomplished increasing the USN on the Exchange 5.5 servers by 101,000. They are still not in synch. our primary Exchange 5.5 Server replicated the same directory information back to all other Exchange 5.5 servers; authrest did not bring them all to the same level as was previously thought.

Options

Currently our primary Exchange 5.5 Server is still acting as the authoritative Directory server in our Exchange organization. Its USN is the highest among all of our Exchange 5.5 servers. Based on this updated information, here are the options that we have in dealing with this situation…

1. Continue to use our primary Exchange 5.5 Server as the master Exchange 5.5 Directory server. All changes to all exchange objects should be made on this server. This includes DL membership modifications and new user mailbox creation. (Mailboxes for other servers can still be created on our primary Exchange 5.5 Server, under the advanced tab of the new mailbox wizard, simply select the home server you want the mailbox created on).
2. Exchange logs an event to the application log during replication of the Exchange 5.5 Directory. We could identify each servers USN level (or number), then based on these results, use authrest to increase each servers USN to a specific number with the goal of individually increasing each other Exchange 5.5 server to thesame number. So authrest would be run with different increment numbers based on that servers existing USN number. Using this method we can correctly use authrest to re-synch the Exchange 5.5 Directory.

I recommend option 1. I dont see any real need to cause further exchange downtime to fix an issue that wont cause us any problems in the Exchange 2003 migration. I can live with running manual backups on our primary Exchange 5.5 Server until its replaced. Backup and restore on our primary Exchange 5.5 Server using NTBACKUP was verified to be working.

Summary Information???

The emergency documentation used to restore the Exchange 5.5 Directory in early March did not include the functional information on authrest that is now known. The attempt that I made to re-synch the Directory was based on the information I had of authrest from the emergency restore that was performed earlier. It took several hours of research to find more information on this tool as it relates to Exchange 5.5. Most of the available information is in reference to an updated version that runs on Exchange 2003. Only passing references to this tool were located. Since support for Exchange 5.5 has ended, its been increasingly difficult to find information on Exchange 5.5.

Continuing to use our primary Exchange 5.5 Server as the authoritative Directory server will not adversely affect the Migration to Exchange 2003. If all changes to Exchange 5.5 objects are made onour primary Exchange 5.5 Server, it will always have the updated and correct Directory information. We can then proceed as planned and use our primary Exchange 5.5 Server in the ADC connector as the source of Exchange 5.5 Directory information to be imported into AD and Exchange 2003. By using option 2 above, any exchange 5.5 servers could be used for the ADC connector, but only 1 server is needed. ???

Once Exchange 2003 is successfully implemented for the first time, it will be much easier to add more servers later on. We can then migrate users and other Exchange 5.5 objects to Exchange 2003. From that point on, most of these issues we have been experiencing lately should be a thing of the past. The new infrastructure will prove to be much more resilient and easier to work with.

NOTE: Changes made on our primary Exchange 5.5 Server may not be immediately visible on the other Exchange 5.5 servers. Replication should take place within 5-15 minutes. A manual directory refresh can be done through the Exchange Administrator by highlighting the exchange server you want to update, double click on the Directory Service, and then click Update Now. Choose the option to update only new or changed items. Then click ok. This will force the remote server to update its Directory.

Thanks for reading! Hope this helps someone!]]>

Home Network – Part 3

Microsoft Active Directory:
My home network is built on Microsoft’s Active Directory. I use active directory to organize my user accounts (all two of them), my computer and group policies. With group policies I can set common variables for all my workstations, servers, etc. This way I don’t have to hand configure everything, its all automatic. Group Policies are a great way to manage your network workstations or servers. There are other solutions here, some people like to run Linux at home, and I’ll admit, I do too from time to time. I love linux, but there are still too many apps I use that require Windows. From time to time I demo some of the latest Linux distributions and try things out. I think its great, and if I had a 4th computer to run it on, I’d probably run a linux server or desktop as well. Some people like novell, some people like MAC, its up to you. This is just how I am doing thing. I have group policies set to add customization to my desktop mainly. Things like a browser title, automatic update settings, common software distribution, etc.

Domains, e-mail and more:
I guess I can’t go much further without explaining how I also do my domain names and websites. I’ll write more about this topic later on as a how to and what you should know for getting your own domain and website. But for now, I’ll keep it simple. I own several domain names which I use for various purposes. I have one domain that is for all my server equipment, like my hosting server that hosts my website and some other websites I host for people (for free unfortunately). These servers are in a data center and I simply “rent” the server from them on a month to month basis because its cheap and does what I want it to do. Plus they take care of maintenance and problems. Then I have a primary domain name I used to use for my hosting company’s website. The backend server domain ended with a .net and the primary domain is a .com. These extensions can be anything you like, but I stuck with a traditional format. Then I have a third domain for my personal website which is mainly for my family and my blog, etc. Here is where the bulk of my incoming and ougoing e-mail comes from, the other two domains are mainly for servers and a now closed hosting company. I do have some other domains, but don’t really used them yet. I’ll be expanding that later on as well.

E-mail:
So now you know I have a shared hosting server which hosts my websites and most functions of my domain names. Now when it comes to e-mail, you’d naturally assume this server also handled mail for my domains as well right? If you said yes, you’d be wrong. I’m using a service called Rollernet which is a mail forwarding service. Since my ISP restricts incoming traffic on port 25, it was necessary to setup SMTP on a non-custom port. However, this causes a problem because when someone on the internet sends me an e-mail, most mail servers only send mail on port 25. So if I’m running SMTP on a non-custom port, how do I get my mail? Here is how. Rollernet’s servers are listed as the MX records for my domains. This means, that when you send me an e-mail, its actually received on port 25 by rollernet. They take the mail, queue it, do some scans on it for viruses, spam etc, then they forward that mail to my home mail server on a custom SMTP port. Of course I have this port setup in my cable modem and firewall to allow it to be forwarded to my mail server which resided on my LAN. Now here is the complicated part. My home mail server received mail on a custom SMTP port and is received by NoSpamToday, which is my SMTP level SPAM filter. NoSpamToday (NST for short), filters for SPAM, viruses etc, and basically makes sure that the message is valid before it allows it in to my mailbox. Now NST is not a mail server, its just a SMTP server, so another component is needed here, thats where 602 Lan Suite (LS for short) comes in. NST received a message for me on a custom SMTP port. Once it makes sure that the message is valid, it then forwards that message to 602LS which receives the message on the standard SMTP Port 25. 602LS receives the message and performes a few checks of its own, like scanning it again for viruses, doing aother SPAM check and finally delivering it to my mailbox. 602LS also has a built in webmail server, so I can check my webmail from anywhere in the world. This is also where port forwarding comes in as the ports for webmail need to be setup to route to my home mail server from the outsite. Using my public DNS zone, I can add a record for webmail to my domain, so I can go to http://webamil.mydomain.com/mail and get to my web interface. This way I don’t have to use DynDNS or any of those services, since my public IP on my cable modem rarely changes. Now if it were to change, I’d have to manually update that in my DNS zone. So watch out for that if your using this scenario. I am aware of it and know what to do, so for me its not a big deal, but if your new to this, don’t set this up and wonder why it breaks 9 months later. Keep an eye on your public IP.

Lets now talk about outgoing mail. I don’t know if your like me, but I find myself in situations at work and abroad where I find that my company network or hotel network restricts SMTP servers to their own servers and won’t let you send mail using your own SMTP configuration. For example, at work I run a simple server monitor that sends alerts. But my company has a firewall in place that limits outgoing SMTP traffic on port 25. Now I bet your wondering where the SMTP component from IIS comes in to the picture from my previous post. Here it is. I am running IIS on my mail server but only the SMPT component. So I setup Microsoft’s SMTP service to listen on a custom port (different from my incoming SMTP port for normal e-mail from Rollernet). This way, I can setup my monitoring server to use my custom SMTP server at home to send the alerts. So in my situation, my monitor program detects a problem with a server in my office, it sends an alert to my home mail server on a custom SMTP port. My SMTP server then relays that message to my shared hosting server which then sends it to the desired recipient on a standard SMTP port. This way, I can use SMTP wherever I am, still get my messages or alerts sent and accomplish my tasks. This custom SMTP service is protected by a username and password and relaying with it is denied. Relaying on NST is also forbidden. Ok, so how about my home PC? Ok, simple, we use outlook on our home PC, so outlook is setup to send/receive mail from 602LS through POP3 and standard SMTP. We send a message from outlook, it is received by my home mail server on port 25, which then forwards that mail to my shared hosting server. Some ISPs also restrict outgoing SMTP traffic, so here you may need to setup a custom port on your public SMTP server and configure your mail server to send all outgoing mail over a “SmartHost” or custom SMTP configuration. My shared hosting server then delivers the mail over standard SMTP to the recipient’s mail server.

So in summary, yes this is a complicated setup, and no it may not be for everyone. But I will say this, there is a degree of pride that goes into setting soemthing like this up. Now I’m a Microsoft Engineer, so I’ve been doing networking for a long time. No this is not the way to go about setting up a business or large company. Obviously I’d recommend using Exchange or more powerful mail servers and betters ISP connections. But if your a techie and want to setup a really cool home network, this guide might just help point you in the right direction.

Other Services:
Lets talk remote access. So how do I manage this home network when I’m not home. Easy, RDP. There are lots of people around that don’t like RDP, its not very secure, and has its issues like any other software or technology. For me however, its perfect. I simply forward port 3389 from my cable modem to my firewall and from my firewall to my PC, I can remotely manage any machine on my home network. Now I took it a step further, and actually setup a custom RDP port on my other machines, like my servers and second desktop. This has the advantage of being easy to individually RDP into any machine on my home network without first having to remote into my home pc and then into another machine. In conjunction with DNS for easy naming, its a snap. All you need to remember is the custom port number for each machine. I only have a few so its no big deal, if you have many machines I’d recommend finding a better way, such as VPN. Through RDP I can remote control, and virtually manage any server or desktop on my home network.

Web management: I also use a program called Remotely Anywhere (www.remotelyanywhere.com). Its a great application that runs as a service on Windows. With it, you can remote control, Transfer files, totally manage all aspects of the machine right from a web browser. Its very robust and powerful, with tons of additional features too numberous to mention. Its one of the best web based remote control/management solutions I know of. This can also be setup on a custom port, so it will need port forwarding configured for it as well.

FTP: I used to have a NAS server with FTP setup so I could FTP directly to my RAID5 storage device. Now that its gone, I don’t really use FTP anymore so I removed it. I use an FTP site on my shared hosting server temporarily if I ever need to send anything through FTP. I can grab it from home later.

Internet Access: Because my cable modem and firewall do NAT, its very easy to provide for internet access to my workstations and servers on my home network. The firewall is the gateway on my network, and Microsof’t DNS handles all DNS related operations on my network. My DNS server is configured to forward all requests for external host names to my ISP’s DNS server. It then caches the results and can reply much faster to any requests my workstations or servers make. Internet access is basically a simple NAT solution provided by my firewall and cable modem.

Points of Failure:
With a system like this there are other considerations that need to be taken into account. Amoung them are power, redundancy, damage, replacement, etc. For example, if my power goes out what happens. Well for me I have my critical equipment on a UPS. Since this is a home network and not a critical system, the UPS will keep my servers and internet connection up and running for 5 minutes. This should be sufficient as long as the power isn’t out for long, which is isn’t usually. What if my firewall or cable modem goes bad. Well then I have a problem, as with my ISP I have to have them come and activate a new cable modem. So I’d first have to buy a replacement and then have them install it. This can be done usually by the next day. So what if my mail server or other network equipment is damaged. Well, for mail, if my home mail server becomes unavailable, mail will queue at rollernet, so I won’t loose any e-mail. I can even redirect that mail to my shared hosting server if I wanted to so I could get to it. If some of my network gear fails, it will obviously need to be replaced. I’d try to repalce it with exactly the same modem so that if it had a configuration with it, I could easily restore a backup config file to immediately get my network back up and running.

Security: What about security, how secure is this setup? Very secure. Even considering I have ports forwarded into my LAN from the outside. This often makes security experts very nervous and for good reason, but again, this is not the NSA, I don’t have anything on my home network worth anything to anyone but me. That is not an excuse for having bad security. First, I have a double NAT solution, so even if someone could hack in past my cable modem, they couldn’t get further than my firewall. If they could get past my firewall by some miracle, they would not be able to access anything on my network, since all network traffic between workstations and server is encrypted through Kerberos. The worst they could do if map out my network and find my IP addresses. DOS attacks are also a possability, but there isn’t much that can be done about that anyway. Again, I’m not saying good security isn’t important, and the measures I’ve taken are sufficient for my needs. Please don’t think I’m advocating bad security measures.

Thanks for taking time to read this post, I know it was long. Keep an eye out for more tech posts in the near future. I’ll also post some images giving you a visual of how all this works. Here is a simple visual aid of what I’m talking about above.]]>