Tag Archive: split DNS

This is the second blog post discussing the change from a .local Domain to another namespace in order to better support the Mac clients in our Domain.

 Planning for the new Domain

The initial decision to move forward with the Domain change was easy compared to what was next.  The next challenge was how and when based on our people resources, major church-wide events, and launching of our 3rd campus.  As far as the life of Clear Creek goes, there are not any very “slow” times in which a project like this would be easy.  The next consideration was the when and how to do this to get the most of our volunteer resources and asking Solerant (IT Contractor).  This is important because while I have some technical skills, I’m really more of a project or resource manager.

  • We decided to deploy on October 9 & 10 because we could start after church services on Sunday and several of our volunteer team could also be present to help on Monday because they had Columbus Day off.
  • We also decided that migrating the Microsoft Exchange Server (E-mail) was going to probably be the most challenging part of the migration and pose the most problems.  I thought that Solerant would be best to do this because they already handle many Microsoft Exchange upgrades each year and that they could handle the added complications that went with a Cross-Domain migration of Exchange.
  • I polled the staff to see if making a huge change like this would seriously impair ministry.  I did my best to explain that everything was going to change, but I don’t know if there is really a way to explain just how much was going to change.  I tried to think of “what the worst case scenario” implications would be to best prepare them.

Once I made the case with my boss and laid out the basic plan we moved forward.  The team normally meets on Monday nights, but over the next few weeks we also met an additional night in order to account for everything that would need to be done.  Sunday the 9th would be the day that the most manpower was needed, so I wanted us to be ready to roll since that was going to be a long day.

Thankfully, in our infrastructure we have an Equallogic SAN and Virtual environment using VMWare.  If we had not had this environment, it would have cost a whole lot more to change our domain at the size we are now. I think we had to add five virtual machines to complete the migration.

Creating a New Domain

The first thing we did was create a new Domain.  We “spun up” a new server in our virtual environment and made it the new Domain Controller.  We then needed to create a “trust” between the new domain and the existing domain.  In order to do that we had to remove the .org from the existing domain’s forward lookup DNS records and setup forwarders to the new Domain’s server.

Extending the Active Directory Schema

This time around, we decided that maybe the Magic Triangle was causing too much delay for the Macs to find resources and that maybe extending the Schema in AD was a better option.  We used Apple’s white paper to change 40 attributes necessary to support Macs in the Domain.  To me, this sounds like a lot, but now I understand that this isn’t in the big picture world of a Domain.

Valuable links:  Apple’s KB and a good blog post, but note that the Dynamic UID problem is not true anymore with a recent patched OSX.  This is how to modify the schema, and the thing we need to do still for applying WGM policies to computer list.

Mimic Necessary Foundation Services

So now we had a domain consisting of a single machine that could talk to the other domain and we could find resources between them.  Now we begin making duplicates of resources that we won’t migrate

  • We created a certificate authority with Web certificate component
  • Network Policy Server (NPS) in order to prepare the way to move our existing Wifi system.
  • We cloned the Print Server and added it to the new Domain which saved huge amount of time with adding drivers and creating shares.
  • Evaluated/Exported all of our Group Policy Objects (GPO) to determine which ones needed to come over, which ones weren’t working, and how we could simplify.
  • Evaluated our Active Directory structure since we had a lot of legacy structure that was not clear, possibly structure from Small Business Server a long time ago.
  • Decided that we would eliminate whatever XP support we could and get the rest of our machines to Windows 7 where possible.  We are currently down to 3 machines with Windows XP with plans to get them out and 5 Vista computers which we will be able to image very soon.
  • Created a new Windows 7 VM and installed the Active Directory Migration Tool (ADMT) from Microsoft on another virtual server joined to new domain.
  • Decided we would migrate Microsoft Exchange as the last step since they were unable to do it beforehand and I only had a short window for my volunteers to help.  In hindsight, it would have probably been best to migrate Exchange FIRST, so that we could migrate the users with their Exchange attributes.
  • We wiped our physical Domain controller and loaded it with Server 2008r2 to match all of our other servers the week before the migration.
  • The morning of our migration, demoted another Domain controller in our .local domain so that we could keep the new Domain pointing to similar DNS to avoid problems with more devices.
  • We made sure we had a second virtual Domain controller on our other Virtual Host server so that we would be prepared if we had any power issues.  We agreed that it would not be a good idea to move a domain controller from one domain to another. It never seems like those things demote properly in my experience. This added a little bit of extra work with re-creating the DHCP server, the options, and his subnets.
  • Joined some test machines as clean, new clients to the new domain including a some Macs: Lion Mini, a Snow Leopard mini, and a Lion Server Mini.  We wanted to get things right with Lion from the start.

What would end user experience during migration?

Since I was not able to quickly find information online about what a user would expect when all of this occurred, I put myself in their shoes in order to make this go for them as smoothly as possible.  My team was busy working on the major aspects of the migration, but they really aren’t wired to think about how the regular person who just wants their computer to do its job might experience with EVERYTHING changing.  I started off with what I know…which is when you add a computer to a domain, there is no user profile.  My assumption became that if you took the same computer from one domain and moved it to another, they would likely have a brand new profile and not see the same desktop as they did the last time they logged in.

  • For most users, this would mean that they would essentially not have anything in “My Documents” or any of their personal settings or see anything they kept on their desktop.
  • Laptops and Macs were going to need extra attention.  The people who use those computers have very specific needs so more is on their specific local machines and thought it would be important to migrate their user data.  We needed tools or scripts for both platforms.  We chose ProfWiz for Windows machines and used these magic commands for Macs.
  • I also knew that because of email, they would essentially be operating as two users – one for email from the existing domain and their other account in the new domain.   I also realized that the regular user was not going to see the distinction.
  • How do I communicate with everyone in a way that doesn’t alienate or frustrate?  I decided to prepare via staff wide email and personal conversations to make sure things were clear to key people.  Then I chose to pick a person in various ministry areas that I could contact to update with the latest information should email be down or we had any major problems.  I could call or text them where to find instructions for them to forward to the people in their area.

How I prepared everyone for the migration

  • I sent out emails to staff asking them to make sure all of their documents were living on our network drives.  We’ve done a pretty good job of having users understand where to store their data.  We choose not to redirect profiles since there has been benefit for many users to put some data on their local computer that can be discarded.
  • People also forget the small things like how to set a default printer or how to export and import their browsing Favorites.  I sent instructions for both in emails ahead of the migration and made sure key people knew how to do those things or print them out so they could help each other.
  • I also sent out instructions to remind users how to report problems by using Solerant’s ticket system or to leave me voicemail message.
  • I made sure that laptop users verified that their data was on the network and for Mac users we made sure they had a full backup in case we had a problem and were not able to help them as quickly as we could.  I’m all for having a backup of your backup.
  • The users also needed to know that they would need to use Webmail (OWA) or email from their cell phones on Monday, until we completed the migration of mail.
  • I made sure I thought about all those “what if’s” on things failing.  My goal?  Have a backup plan to support the end user so they can get their job done no matter what.  We can’t afford for them to not be able to get their job done, nor can we afford to lose credibility with the users because the change was so painful.
  • There isn’t a whole lot on the actual Domain creation and migration that I can do, so I made sure my spreadsheet for all of the servers with their roles was updated and questioned how each one of them would be affected during this change.
  • I created a map of where every machine was physically located and cleaned up the current Active Directory to remove any computers and users that were no longer in service.
  • There were a few other obscure things to remind the team of, making sure the SSO for our RemoteApps were accounted for, KMS setup in the new domain, making sure any references to the Fully Qualified Domain Names (FQDN) references to the old Domain were accounted for as well.
  • During our tests we noticed that our current Domain policy was actually working in that it removed any local user privileges with local administrator rights and made them “guest” users while in the Domain.  To bypass this, I had to make sure my team had good instructions moving from computer to computer on the day of our migration.
  • I also made our Resources team (DVD/CD production of messages) and our Database Reporting Team aware. Both of those teams do not have a staff person in the office during the week, so I needed them to think about how this would affect them and be willing to test to avoid technical issues when they meet or when Sunday comes.


How long did it take?

This is rather tricky to answer, but since I have several people with a little amount of time, I believe it went as well as could be expected with some parts better than others.  It took about three calendar weeks to prepare, one day to move all of the users and computers, and three days to get email moved.  Again, hindsight says we should have moved email first and I still cannot guarantee you that it will be easier.

Moving users and computers

This was a very long day.  We had a good lunch and started after the last church service.  By this time we had made sure we had all of our new infrastructure in place.  I’d already re-imaged some user’s computers during the morning services.  My thought here was that since they were going to have to essentially start over, I may as well give them Windows 7 as well so they don’t have to start over twice within a few months.  There were only a handful of machines I was unable to take care of due to the software or hardware they were running.

  • Six of my volunteers (and me) helped on Sunday and which was kind of tedious process, but these guys were very good.  We worked from 1:30 pm until around 11:00 p.m
  • To start, I had three people working on servers/user migration and the other three working on clients.  It seemed I was more useful answering questions, but I did get to move a few clients.
  • One of those guys made sure we migrated our NOD32 (antivirus) so that clients would see the new server.
  • We also verified DNS for some other servers that were not in the Domain so they would work well after we were done.
  • We moved Ruckus (our Wifi system) pointers to the new Domain NPS (Network Policy Server).
  • We used the ADMT tool to migrate users, had them keep the same Security Identifier (SID), and migrated the password.  The latter created angst with my security-minded volunteers, but I really needed to protect the end user here to prevent distress.  After they migrated the users, I went through each user account and unchecked the box that requires them to change the password and removed the login script since we now push the drive mappings via Group Policy.  I have to say this GPO is fast and beautiful compared to a login script!
  • We moved some servers and workstations using the ADMT tool, but in hindsight I’m not sure that was a good idea for the phone or Remote Desktop Servers.
  • We used ADMT to migrate the file server.  I believe it was very important to move this server this way because it migrated permissions and security groups as well.  Some of folders appear to have had a hiccup and did not duplicate properly, but fixing permissions for a few directories was not as bad as having to start over.   We also consolidated some of our drive mappings to make it easier for our Mac users since they have to browse differently to get to the same shares we make appear magically for the Windows users.
  • Our Church Management system is ACS and we run it via Microsoft’s RemoteApp Remote Desktop Services option  from a Server 2008r2 server.  I created new RemoteApp packages for People Suite and Financials so they could be applied via GPO.
  • Once the servers were done, two of the volunteers started on laptop users, most of which had left their laptops in their offices (our request) to allow us to help them before Monday.  We used a tool called profwiz to migrate user profiles on Windows machines and some magic commands on the Macs to move their profiles.  I wish there was plenty of time to do this for every user, but we just had too many computers.

When we finished, we called it a night.  We had to convert some data LUN’s overnight so that we could completely move to new Veeam backup software (designed for virtual environments).  We left that running while we slept.

Final Blog Post in this series will cover:

Domain Migration:  Challenges & Successes


Domain Migration

This is the first of a series of lengthy blog posts on our recent Domain migration.

About 2 ½ years ago Clear Creek Community church began integrating Macs into our Domain to provide the Arts team with our network resources to attempt to make them equal clients to our resources. The goal was to make our IT resources on mission.

Historical reference and mission orientation

To understand why we did this would require us to go back to the whole church mission and strategy:

  • Lead unchurched people to become fully devoted followers of Christ.
  • Provide multiple campuses of our church body from the Beltway to the Beach and from the Bay to Brazoria County.

My job is to constantly focus on how to accomplish our mission and function like one church body by providing the best technology support to those who directly accomplish the mission.

Clear Creek has grown at a rapid pace which is fantastic for the mission! But, it becomes more challenging to provide for the needs of the staff and volunteers who accomplish our mission.  People need to get their job done in a short amount of time. This is true for everyone in any type of work in general but we feel it more strongly since the majority of our staff are part time or gifted volunteers with a heart for the mission. This growth also highlights how important it is for the team to be unified and cohesive. I strongly felt the need for the Technical and Creative Arts teams to be able to bring their resources together into the same cohesive integration and organization rather than leaving them outside to fend and manage for themselves.

It would be a change in operations for them but I could see how it would help make the team more cohesive and integrated on mission. They also liked the idea of organization and integration. The people hurdle done the technology hurdle remains. Computers just do what they told (no matter what we think when they seemingly misbehave) so we just need to tell them what to do…

Integrating Windows and Apples

Plenty of organizations have integrated the Apple Mac OS X systems into their corporate environment with success, just as many have attempted and failed. My volunteer base is not full of people who know how to support a mixed network. It’s not a very common task. Most business environments choose for the user what tools they get to use.  For us, like many small or large environments, it still makes sense for most users to use a Windows machine because of the business type tools we have chosen to do the work.  Music and video professionals still prefer Macs to get their work done in the best possible way.  Therefore, the arts teams had both a Windows machine and a Mac, which I’ve always felt was a waste of resources.  There had to be a better way to help them.

The push-back in most IT shops, including churches is that Macs are too expensive and support for them takes longer since they aren’t designed for business/work environments.  We also get the push-back from users who prefer the Apple over Windows, because that is what they are comfortable with.   I mention this, because once you begin adding Macs to your network, more people are going to want them.  This will be a balance:  providing the tools people want or need with the boundaries of good stewardship, without alienating the end user because they think we are mean.

There is no “Best Practices” really when you support multiple operating systems.  You can accomplish this by one of three methods:  3rd party per client product, Dual Directory method, or extending the schema in Microsoft’s Active Directory.  It really takes analysis of the tools and the structure of the organization and a lot of trial and error.  We have taken note of where many churches have jumped around with multiple 3rd party tools or completely removed Macs from their domain altogether.  We still think we can find a way to make the Mac user’s machines happy and live well among us.

We first integrated our Macs within our domain using the Dual Directory method, otherwise known as the Magic Triangle.  We do subscribe to a Mac Enterprise users group where we have learned how many schools and businesses handle Macs, and they use one of those three ways.

We ran into issues immediately when Snow Leopard came out.  This added issues for Macs on our network and caused them to take forever to login and connect to resources.  It was even more painful if you had a laptop and logged in outside of our network.  This wasn’t too bad for just a few Mac users, since most of them were desktops and the mobile users were willing to put up with these shortcomings.

Then we had some other key users switch to a Mac and it became evident that this problem was going to be worse.  So a year into our integration, we researched and learned that the problem could be because our internal Domain was a .local namespace.  Push came to shove so I brought up the discussion of getting rid of the split DNS.  We agreed at that time that while that sounded like a good idea, there was really no way to know whether or not this would make things significantly better for Mac users, nor provide any benefit to the Windows users.  Since our network has grown to 100+ PC’s, 20+ servers, 18+ Macs, we dropped the idea.

What is the issue with .local and Mac OS X?

This is a challenge to explain but this is the answer that finally made sense to me.  It turned out I really just needed a good picture of how a Mac functions.  Think of it this way…A Mac is designed “to just work” which is wonderful if you are an end user living on an island of your own Mac.  You turn on your Mac and since it’s “name” always ends in .local, it goes and looks for anything with that name in it to be friends with in case you want to share iTunes or anything else with them.

This isn’t so great in a mixed network with a .local namespace. A domain can’t belong to two masters; it can’t be the fun iTunes Library sharing environment and the DNS served business environment.

When you login to a .local domain that uses the Dual Directory method, The Mac does what it knows how to do and uses Multicast DNS to ask all the machines around it if YOU are the one that knows what it is supposed to do…and it doesn’t seem to really listen since there is too many things for it to check through.  Then, once it finds the Domain, it will login.  It takes about 3 minutes for this to happen on site and about 5 minutes or more away from our network, in order to see your desktop and be ready for you to work.

Time to change

It became clear that moving away from the .local namespace was important.  We still couldn’t actually say what this would buy for us since making this change was going to affect every device, every server, and every USER.  The last was my biggest concern.  Was it worth disrupting EVERY user to make the world for Macs better?  We decided it was worth it to try to help those Mac users before we get any larger and before we get to the point that we serve our church across multiple campuses with the same resources.  Currently, we have multiple campuses, but we office out of the same location.

In the recent past the benefits did not seem to outweigh the risks but this year we gained some new dedicated volunteers:  one that supports a mixed environment like us and another one that has the same Windows infrastructure in his work environment as we do. With the solid foundation suddenly so much improved it was time to push the issue again.  My questions were:

  • Why does having split DNS with a .local namespace present such an issue and is it always going to be an issue?
  • What does this do now that Lion has been released which has its own connectivity issues with Windows and there isn’t a supportable way to downgrade newer machines to Snow Leopard to manage our growing need for Macs?

Now, with a Lion breathing down our necks, the benefits outran the risks.

The next blog posts will cover: