This is the third post about our recent Domain Migration.  If you happened upon this without reading the first two posts, you can start at this link.

Monday, supporting users in the new Domain…

Three of our volunteers showed up bright and early to prepare for any fallout that we encountered.  Dustin from Solerant arrived while I picked up Kolaches (man-food type pasteries) and began the process to move our Exchange 2010 server to the new domain.  Unfortunately, we tried to help him by spinning up a new server and installing Exchange Server which he had to delete and start over.  I should have just spun up and patched a brand new server and stopped there.

Resulting Challenges

  • We had some had hiccups with the data conversion of some LUN’s, and because we later learned that our SAN switches were not correctly configured and Jumbo Frames aren’t enabled, which we will do during our next SAN firmware update.
  • Since the file server was down, the GPO that pushed the RemoteApps were not able to install the ACS RemoteApps until we moved the pointer to another location—which makes more sense in the end. We now share the default location where the Microsoft Packager will create the package.
  • We had some issues with Group Policiesin the new domain. Perhaps they are default policies designed for enhanced security but they did create some issues for us. Among them,
    • Our Windows users still need to be local administrators, but a GPO was preventing that.
    • We have two wide format printers that we do not include in our Print Server since one of them we know did not do well over the network.  The older one was in use all the time, but went out on them while we were making these changes.  The newer one would not print at all, which was odd.  We had not printed to it since July so it was hard to pin down what the problem was.  It turns out the jobs would be deleted almost as soon as it entered the print monitor to print.  The printer tech told me it was a switch, but somehow I guessed it was a GPO, and it was a radio button that wouldn’t allow printing to other printers.
    • The Phone Server caused us some problems because of our own Group Policy removing local administrator rights when we removed the server from the domain initially.   We didn’t notice anything really until we broke the Trust between Domains and shut the old domain on Friday after email successfully moved.  Once the trust was broken, that is when voicemail stopped forwarding to email, which everyone missed right away.  One of my volunteers worked on this the following Sunday, and broke voicemail delivery entirely which I didn’t notice until a few days later.  It turns out that the phone server needed the service accounts to have local administrator rights as well as a new backup software that had been installed on that server that was also interfering.  Voicemail will deliver now and it will also forward to gmail, so it is only a matter of time until we figure out how to get it to forward to our email.  This server uses its own SMTP to handle email.  I left for vacation before figuring this last part out.  I don’t understand phone systems at all, so am thankful that I have enough troubleshooting skills to get as far as I did.
    • The copiers scan to folder function quit working because we forgot to migrate that AD service account and I also fixed the DNS and domain name pointers that included one of the old Domain controllers.
    • End users handled everything with much grace!  Many of them needed help with email profiles being removed and re-added through the control panel.  Group policy didn’t always lay down the first time the user logged in, but helped them quickly with the gpupdate /force command.  Vista computers kept both sets of printers, from the old domain and the new domain.  Since I only have a few, I just deleted the old printers.
    • There was also a day when Exchange had a few services quit on us, but that hasn’t occurred since.  That distressed a few users that Friday, but we were able to rectify that with Solerant’s help rather quickly.
    • There were some changes we needed to make with our Proxy since it could resolve incorrectly off site.  Once we made the change we were able to help users with laptops and the Mac users.
    • Exchange didn’t get completely migrated until the 4th day due to problems.  I was able to offer workarounds for users via webmail for Monday and Tuesday, and our list serve or Constant Contact for mass mail. Dustin’s problems began because we migrated user accounts to the new Domain and included the Exchange attributes.  When he did get to where he could move mailboxes several kept “failing”.  Perhaps it was because we had the SAN switches configured incorrectly and it was taking too long to move mailboxes.  Dustin did get help from some of his colleagues and some answers from Microsoft.  In the end all of the “failed” mailboxes were there so nobody lost any email.  There was the oddity of “duplicate” users which he deleted using the Exchange the Command Shell.  One of my volunteers and I were able to work as a team with Dustin on Thursday morning to reconnect each user with their mailbox and release our spooled mail from SpamSoap.  No incoming mail was lost during that time.  It was a wonderful sound to hear people’s phones and Outlook begin chiming that their mail was being delivered.
    • With all the changes we made, we had some issues supporting Windows XP.  Rather than make the changes we would need to make in security to help them, we will help them upgrade to Windows 7.  Microsoft XP has a different way of handling Kerberos that is less secure.  For instance, drives would map, but they are unable to get to the network shares.

Problems caused because Microsoft Exchange was moved last

The ADMT was perfect for migrating users and some computers, but it should not have been used to move users before the Exchange Server was ready.  I wish there had been more lead time for the project so this could have happened better.  The other truth is that this is something not often done and not many systems engineers will ever do this kind of project.  We made it through and overall it went well.  I just like to look for how things could have been better so that with whatever next project we complete we can find more potential problems ahead of time.

  • Because we moved users and computers first AND we migrated Exchange attributes, Dustin had to remove those attributes and re-migrate all of those users a second time.
  • While Exchange was migrating, the duplicate users in the new domain REALLY confused the Macs which had been working beautifully.  Mac users were unable to login at all if they had rebooted from the night before.  If they had not been shut down or if they had been off site, they could login and work.  I was able to pinpoint the problem’s relation to the second user account and my Solerant tech was able find a temporary workaround for users by using an account that didn’t have a mailbox attached to it.
  • The Remote Desktop Servers all had triple profiles for users (one for the .local Domain and two for the new).  I suspect this may be related to the same issue above compounded by the old domain.
  • This prevented successful launching of Outlook and autoconfiguring their mail to those profiles.  My solution here was to just remove all of the user profiles from those servers and let them start over to prevent less intervention on my part.  Some users with access to different tools still needed a little assistance, but this wasn’t a horrible problem, just a little time.

The Success we have seen already

Something we noticed immediately is how quickly a Windows user can get from the login screen to their desktop.  GPO’s apply very quickly once the user has a profile on that machine and I really like the new drive mapping GPO.

Mac users immediately noticed a difference.  Many Mac users have told me how much faster login is. That first week the Mac user with the oldest laptop told me how he loved how quickly he could login, both on or off site.  The new Mac OS X share connecting login script also applies quickly.

All of our Group Policy objects apply whether we like it or not.  Okay, that last part was for humor since we told it to do it to begin with.  This was great for cleaning up our Policies and simplifying things the best we can, a thing difficult to accomplish the larger and more complex we get.

More of my volunteers have a thorough understanding of what they support apart from the networks they work with for their full-time jobs.  I hope this helps us as we grow into more campuses and provide the technology needed to continue to further our mission.

Really, this project went well overall.  To be able to find the problems in spite of all of the changes that occurred at the same time is truly miraculous, I believe. I am thankful for my volunteers who helped:  Kevin Creason, Michael Huset, Paul Salvo, who spent the most time preparing for migration, day of migration, and helped with fallout afterward.  I am also thankful for Dustin Corkern, the Solerant Engineer who stuck with the email migration until it was done.  Thanks also to Paul Rhodes, Donald Cook and Jeff Ammerman who were able to help when they could before and on the day of migration.

Advertisements