blogs.conchango.com

So, my first proper post...

I've recently been working on a way to take a running virtual machine, that has a full, single machine installation of BizTalk Server 2004 (with Sharepoint Integration and everything else), and to clone it so that virtual developer workstations can be provisioned with minimal fuss and hassle. Neat idea I hear you say, so why the blog post? Well, this isn't a straightforward task, as I'm sure you can imagine...

Background

My current client operates an integration development model where a central team builds, maintains and enhances a shared integration infrastructure, which provides a service to application teams. The application teams design and build orchestrations using the pipelines, visual studio templates, etc. This client also has a fairly complex Active Directory set up, the long and short of this is that the machines that developers use to build BizTalk solutions are in a different domain to the 'normal' (ie .NET) development machines, meaning that a) we can't just wack BizTalk onto their current machine, and b) we have to provision a developer machine relativly often. Also this client has made an extensive investment in Virtualisation technology in the form of VMWare ESX Server, so in theory we can just clone a running machine with BizTalk on and ba-da-bing we have a new developer workstation, right? Read on to find out how much hassle this really was, and for pointers on how to do it yourself...

Theory

Cloning a virtual machine is pretty simple stuff. If you use Virtual PC, basically all you have to do is copy the folder containing the virtual harddrive and vmc file, rename the vmc file, and double click the vmc to add it to virtual PC console. Voila, you now have two virtual pc's with exactly the same config. Easy? Well no. The trouble is that the config is too much the same. You can't run both at the same time (unless you isolate them from each other) because they'll conflict with each other on the network. Not only do they have the same machine name, but more importantly, underlying both the machine identity and each local user account & group on the machine is a Security Identifier (SID) that must be unique, but is obviously not once you copy the virtual machine.

So what is described above is not cloning, but copying. To clone you need to do what OEM's like Dell do, and use Microsoft System Preparation Tool (sysprep). This is included on your OS cd as part of the deploy.cab file, tucked away in the <root>\support\tools folder. Now there's a lot to sysprep, and automated deployment, more of which can be found here, but basically you take a working system, create a config file for sysprep, run sysprep (which culminates in the machine shutting iteself down) copy the harddrive, (in a virtual world you copy the file that is the harddrive for the vm, whereas OEM guys copy the physical harddrive contents from one PC to another.) And when you next boot from that harddrive, Windows will prompt for a new name for the 'clone', and importantly will change all the SIDs to new ones, plus will allow all maner of settings to be changed from the admin password to the locale, either prompting the user for these, or these new parameters can be passed to sysprep in a configuration file.

Does booting a PC and entering a load of personalisation sound familiar? Well anyone who's bought a PC with the OS already installed will have gone through this sysprep 'mini-setup' at the first time of booting...

So, that's cloning, big deal, could have found out about that anywhere, now why the blog post?

OK, sysprep is only really designed to clone the OS. Applications (like BizTalk, in fact especially BizTalk) store values that relate to the old machines identity (such as the machine name, in fact mostly the machine name, but the change in SID is also very disruptive to SQL Server logins) in places that sysprep can't know about (as it's an OS tool), and therefore can't update. So once you've run configframework the above process no longer works. That is to say that the OS clones fine, but BizTalk, Enterprise Single Sign On, Windows Sharepoint Services, to a lesser extent SQL Server, and a whole host of other stuff will not be happy bunnies.

Help is at hand

What I've done, therefore is to make use of the ability for sysprep to run one or more configurable commands once it completes, and to have it call a batch script that does a whole heap of stuff to 'fix' the clone. Obviously I can't give you the code, but I can point you towards what needs to be done:

1. There are many locations within the databases used by BizTalk where the old server’s name is hard-coded. I wrote a T-SQL script that searches through the BizTalk databases and replaces any instances of the old server’s name with the new server’s name.
2. The same is applied to the Sharepoint databases
3. SQL Server Analysis Services uses an mdb file to store its configuration, including the instance name of the SQL Server. I use some VBScript that opens the mdb and replaces the old entries in this file.
4. There are many, many places where the old machine name is hard coded in the registry. I painstakingly searched through the whole registry to find each entry, then created a template reg file of all these entries with an identifier instead of the machine name. Then at fix time i use a small utility from the Windows NT 4.0 Server Resource Kit called munge.exe to do a find-and-replace on the template replacing the identifier with the new machine name, then I import this newly created reg file.
5. Also, the performance monitor counter entries are stored in the registry as hex values for some reason. I use a VBScript to generate another Registry File that updates the performance monitor counters registry entries.
6. The BizTalk BAM components store the server name in a configuration XML file.Again I use munge.exe to fix this.
7. This next one caused me a lot of grief. Microsoft: *please* change sysprep so that it is IIS Metabase aware..! Anyway, IIS stores its configuration in the IIS Metabase. This is a hierarchical structure much like the registry. Each entry can have permissions set using an Access Control List (ACL) which grants or denies specific access permissions to specific users. The ACLs in this configuration store are not updated by Sysprep. I use some VBScript to update these ACLs.
8. The Registry also allows Permissions to be set on each key in the form of a DACL (Discretionary ACL, if anyone knows the difference between a DACL and an ACL, let me know!). These are also not updated by Sysprep (although IMHO certainly should be, again, Microsoft: *please* change sysprep so that it updates registry ACLs...) This step updates these using a neat little utility called regperm.exe. *Update 6/9/2006* I've found a better way.
9. I was using VMWare, not Microsoft Virtual PC or Virtual Server, as this is what my client has. VMWare drives the sysprep process for you, and for some reason, VMWare does not seem to honour the configuration parameters that are used to set the locale of the server when cloning. This may be a configuration issue to do with my clients installation thoug, YMMV. Anyway, due to a bug in the BAM implementation (acknowledged by Microsoft) the machine must have its locale set to ‘English (US)’ in order to be able to deploy BAM solutions. As the VMWare host is in the UK, and set to 'English (UK)', after cloning, VMWare automatically sets the locale to ‘English (UK)’ regardless of the configuration set through the cloning process. I have to use a bit of trickery, found here (ignore most of the article, it's the command about two thirds down that does the trick) to change the locale back to ‘English (US)’
10.  Changing the locale requires a reboot to take effect. Therefore I ,make autologon settings and runonce settings in the registry that will cause the server to automatically log on as the local Administrator account after the server reboots, and continue the clone process by executing a follow on batch script. Then I use shutdown.exe that is part of Windows Server 2003 to reboot the computer.
11. Obviously, after the Server reboots and automatically logs on, I remove the registry settings that cause the automatic logon.
12. After cloning the server any BAM deployments that had previously been done have to be un-deployed and re-deployed to restore them to functionality. I scripted this in a batch file.
13. Because these BAM deployments are done by the local Administrator, their associated SQL Server Agent jobs are owned by the local Administrator. My clients security policy is to remove SA privelages from the local Administrators group. Once this happens (at the end of the cloning fix up proces), these jobs will fail to run. Therefore I use a bit of T-SQL to change the job ownership for all of the SQL Server Agent jobs so that they are owned by the service account that SQL Server runs under.
14. And apart from removing the SA privelages from the local Administrators group, and cleaning up, that's it!

Caveats

This works for me for one reason, and this is one big reason why it may not work for you: All of the service accounts (the accounts used for IIS Application Pool identities, to run BizTalk, Sharepoint, SQL and EntSSO services, etc) are domain accounts.

What this means is that when syspreping, these accounts SIDs do not change, as sysprep only needs to change the SIDs for the local security accounts, groups and the machine. If you are using local accounts to run BizTalk etc under then when the SIDs change you will have a whole extra problem to fix: SQL Server identifies logons that use Windows Authentication by the SID. So when sysprep changes the SID, it obviously doesn't update SQL Server as it's not aware of server software, then when 'My Service Account' that is the identity for say a BizTalk host tries to login to SQL Server, SQL Server says, "you're not 'My Service Account', you have a different SID to the one I've got stored here, go away!" By using domain service accounts this problem is removed.

"But I just want to clone a standalone VPC on my laptop" I hear you say. Well, you can fix the SQL Server logins, but luckily I didn't have to ;-), because it's not a trivial exercise... I'll leave this as an exercise for the reader! BTW, if you do work on this, please let me know how you get on :-)

Conclusion

After each aspect above is fixed, the macine is fully restored to perfect functioning state and does not even require a further reboot! Machines that have been cloned like this have been used without a single hitch at my client for several weeks now. Plus you can even still use Visual Studio to develop ordinary Windows Forms, ASP.NET, and Web Service applications (as long as you beat up sharepoint to stop it messing with IIS, but that you would have to do even if it wasn't a clone. I think that may be another blog post...!)

Jack