James Saull's Blog

SQL Server Enterprise Edition. Not all indexes can be re-built online.

James.Saull — Fri, 08 Aug 2008 16:25:25 GMT

When designing your application schema you need to spend some time planning for operations. You may plan to rely on the Enterprise Edition of SQL Server to bring you online index rebuilding to ensure that index maintenance does not require downtime. The excerpt below from Microsoft SQL Server Books Online is important to remember because if you choose certain datatypes then online index rebuilds may not be available to you:

Clustered indexes must be created, rebuilt, or dropped offline when the underlying table contains large object (LOB) data types: image, ntext, text, varchar(max), nvarchar(max), varbinary(max), and xml.

Nonunique nonclustered indexes can be created online when the table contains LOB data types but none of these columns are used in the index definition as either key or nonkey (included) columns. Nonclustered indexes defined with LOB data type columns must be created or rebuilt offline.

Indexes on local temp tables cannot be created, rebuilt, or dropped online. This restriction does not apply to indexes on global temp tables.

http://technet.microsoft.com/en-us/library/ms190981.aspx

Just a warning, in case you are planning for a very high availability solution.

Windows Home Server, the iPod and the XBox

James.Saull — Tue, 05 Aug 2008 08:56:00 GMT

With all my music safely stored on the Windows Home Server (and Amazon S3 courtesy of Jungle Disk) I installed the Home Server client software on the wife's laptop (so it would get backed up). I mapped a drive the Music share and then told iTunes to "Add a folder". I pointed it at the mapped drive and that was that. I was quite surprised just how long iTunes took to "add the folder". It appears to scan all the file names, metadata and probably a whole lot more - either way it is very intensive on the network and takes far longer than you'd expect. This makes it annoying that it is not very multithreaded. Of course adding music to this folder other than through iTunes means that, by the looks of it, we have to keep telling it to add this folder. It is not very good at automatically keeping up to date. Seriously - what is the fuss about iTunes? it is a ~70MB fat client app that seems to struggle with the basics. Importantly though, iTunes knows about our WHS music repository and can therefore create play lists and sync them to the iPod. Success with gripes!

I then fired up the XBox, navigated to the "Media" tab and chose to add a media source. It had already detected my Windows Home Server. That was it. I then clicked on the "Albums" list and it thought about it momentarily before displaying all the albums. The experience was the same when selecting "Artists". Either way - significantly faster than iTunes. What is quite nice is that whilst it is playing an album you can go back to the "Media" tab and navigate to the photos stored on the Windows Home Server and kick off a slide show of a folder and all its sub-folders. It was nice to have the XBox playing music whilst slide showing all the pictures of the kids.

I haven't tried to show video content on the XBox yet.

When is a PowerShell script not really a script anymore? Why not just code it as a C# command line application?

James.Saull — Mon, 04 Aug 2008 20:14:40 GMT

After all, can't I just use notepad and csc.exe as long as I can remember "public static void main"...

First of all, why was it written as a script in the first place? What were the perceived advantages? After all, just because you can doesn't mean you should!

Transparency: scripts can be opened in a text editor even in the production environment. Systems administrators can see what it is doing. They aren't the sort of people familiar with Reflector, and they probably use PowerShell for many of their tasks anyway.
Debug: a corollary of transparency. If a script starts to fail in production it can be easy to debug and modify a copy of that script right there in the production environment, and not necessarily by the original developer. I don't just mean by printing the output to screen - stepping through style debugging.
Modifiable: contentious this one. A system administrator can modify the script as environments change. This is very quick and easy to do without a full-bore compile/deploy cycle from a development team. Agreed, many of the things likely to change will be in a configuration file (you could argue that is all a script is, so why have yet another text file unless it is shared). However, if the script is being modified in the deployed environment then it is not part of the solution it was originally deployed with anymore - this could be remedied with a sleight of configuration management though.
Purpose: with all the CmdLets available in the base deployment or the Community Extensions or PowerPacks etc. you get some enormous re-use to do the sorts of tasks that it was intended for: managing files, working with the contents of files and objects flowing down a pipeline.

Scripts can be well written feats of engineering: aesthetic, efficient, well factored, readable, maintainable, instrumented, documented, following a common convention/style, deployable etc. I think the issue gets tough when the coding gets complex and starts using a great deal of .Net BCL to get the job done. Sometimes it makes sense to move this code into CmdLets or Plain Old C# Objects (POCO) which you can manage in a more rigorous development practice - critically: unit tested and strongly typed where it helps. These functions we have to feel more comfortable wrapping up (inside Unit Tests, Code Coverage, Static Analysis, FxCop, StyleCop, NDepends etc.) and gaining the purpose that is clearly better expressed within an object oriented .Net project at the expense of the transparency, modifiability and general "Operations" friendliness. I would definitely feel better with a 1000 lines of unit tested C# expressed as objects, interfaces and so forth. It's why the PowerShell folk gave us the continuum from the one-liner, through scripts to CmdLets and even custom hosts etc. The trick is to spot where you are on that continuum and refactor as you move along it; recognising the trade offs and benefits.

When I see a PowerShell script that is managing variables and composed of simple routines and CmdLets fulfilling a simple role I feel comfortable; but when I see hundreds of lines of PowerShell mostly made up of calls into the BCL it makes me feel as if it has lost its way and is perhaps better served being moved into CmdLets or Plain Old C# Objects developed in a traditional C#/Visual Studio environment. I don't want to see PowerShell used as an excuse to slap some script together, and move away from the rigour of our engineering practices.

What do you think defines a script and what defines a C# Command Line EXE / DLL / CmdLet? Is it complexity? Who will be maintaining it? The programming style? Simplicity vs Complexity? Is it the reuse of blending the CmdLets that brings speed and efficiency to a certain set of tasks?

PowerShell + VMWare

James.Saull — Tue, 29 Jul 2008 13:18:34 GMT

Our development endeavours often include many tens of developers each with their own managed virtualised development environment. Typically, on top of these, there are multiple test and pre-production environments (each with clusters or load balanced clusters). We can therefore easily end up with 50+ Virtual Machines on a single project. The development/test cycle is very fluid creating atypical management demands on the environment. It is fair to say that these environments easily pass the threshold where scripting management routines becomes worthwhile. To help with this, VMWare have released a whole host of PowerShell CmdLets to wrap up the management web services. If it has got PowerShell in it, it gets my vote!

So here are the VMWare PowerShell libraries:

http://www.vmware.com/sdk/vitk_win/index.html

http://www.ntpro.nl/blog/archives/561-VMware-Infrastructure-Update-Manager-PowerShell-Library.html

Here is a blog covering the VMWare Infrastructure Toolkit (VI as they call it - sounds like my favourite text editor):

http://blogs.vmware.com/vipowershell/

And here is a book on it:

http://halr9000.com/article/541

In fact I just noticed PowerGUI integration to VI Client:

Windows Home Server Power Pack 1 and WHS+USB 1.0 == Misery!

James.Saull — Wed, 23 Jul 2008 20:39:11 GMT

I pushed too much data on to the Windows Home Server (WHS) and I started to get health warnings indicating which files it could no longer duplicate. My data was stored OK, just at risk. Given that the WHS experience was going well I ordered up a Lacie 500GB external USB 2 drive. To tide me over, I plugged in a little 100GB host powered USB drive and added it to the pool. Slowly the data balanced out across the three drives with enough space for all content to be duplicated. I then continued to copy more data on to the server. Transfer rates were abysmal! 500k/s. Oh dear. All my content finally copied over and eventually my critical health warning went away as it finally migrated enough data off the smallest system drive.

The Lacie 500GB drive arrives and I plug it in. A couple of clicks later it is added to the storage pool. It starts to balance the content whilst I right click the little 100GB host powered drive and remove it from the storage pool. WHS can see that there is enough storage and drives to successfully duplicate content across the remaining drives and so it starts to migrate content off... how slow?! it can only have about 30GB on it. Max! Anyway, time for bed.

By the next morning all the content has migrated off the drive. I go to the WHS to remove the drive... Ah... I'd forgotten that this old machine still had USB 1.0 ports. In the darkness under the desk it would appear that I had rediscovered them when I had plugged in the 100GB drive! Copying files and balancing storage with the 500GB drive plugged into USB 2.0 is now going up to 8MB/s (given I have a 10/100 switch in the way it won't get much higher than that). When MS say they don't recommend USB 1.0 they weren't kidding. Copying large AVCHD files onto the WHS is nice and fast and I can happily watch the HiDef videos directly off the WHS.

Applied WHS Power Pack 1 last night. All straight forward. Go get it here: http://download.microsoft.com/download/8/b/d/8bd7fe82-34cc-4026-b8e5-e9e10902312c/WHS-KB944289-v1-x86-ENU.exe

and read about it here: http://blogs.technet.com/homeserver/archive/2008/07/21/power-pack-1-come-and-get-it.aspx

Windows Home Server

James.Saull — Tue, 15 Jul 2008 12:03:24 GMT

I successfully installed Windows Home Server on a 7 year old Athlon 1800 with 512MB of RAM and two ATA drives (100GB and 200GB). My first concerns were: would the BIOS support booting from the DVD drive and was it a USB 2 card in there? Thankfully yes on both counts. One day I will save up for one of these lovely machines from tranquil-pc: http://www.tranquilpc-shop.co.uk/acatalog/HOME_SERVERS.html. Atom power devices with Terabytes of internal storage, expandability and very low power consumption - think half a light bulb, and even then they can be sent to sleep during the day when you are out of the house. But for the time being, I am extending the life of my home built tower...

It performed a large number of reboots and the aged machine took a long time to get through the install so when it got to applying updates I switched off the screen and went to bed. All very uneventful.

Originally I was using Robocopy scripts to mirror data between the two drives. I had to copy this data off on to a USB drive because installing Windows Home Server formats all drives.

Next steps:

Selectively bring back data and put it into a new folder structure.
Install the WHS client on wife's laptop and hope that iTunes is happy to use the shares as libraries!
Install Jungle Disk for Windows Home Server to backup my data into the cloud.
See whether I can get Windows Media Center to happily consume content from Windows Home Server.
See whether I can get XBOX 360 to consume content from Windows Home Server.
Buy a sizeable external HDD so there is plenty of space to back up the laptops.
Configure backups of my laptop and wife's laptop - AC powered, but hibernated/sleeped laptops should wake-up and backup in the middle of the night.

A useful link for Windows Home Server addins: http://www.wegotserved.co.uk/windows-home-server-add-ins/

At the time, did you know it was the last time?

James.Saull — Mon, 16 Jun 2008 21:50:20 GMT

As a parent you watch your kids go through phases, just you aren't always aware that you are entering a phase or leaving one. For example, when is the last time you actually change a nappy? When was the last time you used the buggy to cart them around? You probably don't realise it at the time. One day you do it not realising it, and then the next day you forget to take the buggy out, then you realise that you didn't really need it and then you don't bother... unwittingly that day was the last day.

When was the last moment I wrote a line of Korn Shell, PERL, Java or VB6? Once upon a time those languages dominated my daily life, and then one day, unknowingly, it was the last day. When will it be the last day that I actually turn a line of C#? Will it be because my colleagues can't take it anymore? Or will it be because, by chance, I find myself on an engagement that forces me to use some special functional programming language to solve a massively parallel computing task and then I never end up with the opportunity to use C# again? Who knows. I wonder what that line of C# will look like? Probably just a closing brace and I won't even know it.

NUDA

James.Saull — Wed, 04 Jun 2008 22:43:54 GMT

With plenty of people anticipating the massively multiple core CPU era, NUMA architectures in regular workstations, and all the wonderful developments in parallel extensions to development languages and software transactional memory to take advantage of them; I see one big bottleneck that looks set to defeat most of this on regular desktop computing: disk IO. My laptop is dual core and is always disk bound. Is disk going to keep up? How long before I can reasonably (just like dual core is very reasonable in a laptop) have multiple large solid state disks in my laptop?

Will they develop NUDA? Non Uniform Disk Architecture? In other words a disk subsystem that I don't have to micro manage that dovetails into NUMA. Once memory is exhausted the OS has to go to pagefile, but that page file should be held on striped solid state disks, and when that is exhausted it should be on striped hard disks and eventually very large storage on non-striped large disks. Or perhaps more like a continuation of NUMA where it would not stripe the disks but instead devote solid state disk volumes to each memory bank so that it's page file is closest and not competed for. That might not work though: if a thread and memory got migrated to a different set of cores and memory it would have to migrate disk data too which would be very bad. Better to stick with shared storage and striping.

Ramble Ramble.

Invoke-RemoteCommand - PowerShell CmdLet to tide me over until PowerShell V2

James.Saull — Wed, 04 Jun 2008 22:22:02 GMT

To tide me over until PowerShell V2 ships and we can perform remote PowerShell out of the box, I have created a simple WMI based CmdLet that will asynchronously execute a command - remotely. You might find it useful too.

The main limitation is that it does not track the result of the remote execution. You should be able to call any .EXE such as "cmd.exe /c" or "powershell.exe -command".

Here is the C# snippet - I don't code much so use with caution!

using System;
using System.Collections;
using System.Collections.Generic;
using System.ComponentModel;
using System.Diagnostics;
using System.IO;
using System.Management;
using System.Management.Automation;
using Microsoft.PowerShell.Commands;

namespace JamesSaull.PowerShell
{
    /// <summary>
    /// InvokeRemoteCommand is a PowerShell CmdLet using WMI to execute a command on a remote server.
    /// </summary>
    /// <example>Invoke-RemoteCommand -TargetServerName WEB01 -RemoteCommand "cmd.exe /c mkdir c:\temp\testfolder"</example>
    [Cmdlet("Invoke", "RemoteCommand")]
    public class InvokeRemoteCommand : DCPCommandLet
    {
        #region Fields
        private string remoteCommand;
        private string targetServerName;
        private string username = string.Empty;
        private string password = string.Empty;
        #endregion
        #region Command Arguments / Public Properties
        /// <summary>
        /// Name of the server where the command should be executed.
        /// </summary>
        [Parameter(Position = 0, HelpMessage = "Name of the server where the command should be executed.", Mandatory = true)]
        [ValidateNotNullOrEmpty]
        public string TargetServerName
        {
            get { return targetServerName; }
            set { targetServerName = value; }
        }
        /// <summary>
        /// The command to be executed. e.g. cmd.exe /c mkdir c:\temp\testfolder
        /// </summary>
        [Parameter(Position = 1, HelpMessage = "The command to be executed. e.g. cmd.exe /c mkdir c:\temp\testfolder", Mandatory = true)]
        [ValidateNotNullOrEmpty]
        public string RemoteCommand
        {
            get { return remoteCommand; }
            set { remoteCommand = value; }
        }
        /// <summary>
        /// Username used to authenticate and authorise against Target Server. Not allowed for local connections.
        /// </summary>
        [Parameter(Position = 3, HelpMessage = "Username used to authenticate and authorise against Target Server. Not allowed for local connections.")]
        [ValidateNotNullOrEmpty]
        public string Username
        {
            get { return username; }
            set { username = value; }
        }
        /// <summary>
        /// Password used to authenticate and authorise against Target Server.
        /// </summary>
        [Parameter(Position = 4, HelpMessage = "Password used to authenticate and authorise against Target Server.")]
        [ValidateNotNullOrEmpty]
        public string Password
        {
            get { return password; }
            set { password = value; }
        }
        #endregion
        #region Overriding CmdLet
        /// <summary>
        /// Currently this CmdLet does not allow usage in a pipeline so it only overrides EndProcessing and not ProcessRequest or BeginProcessing.
        /// </summary>
        protected override void EndProcessing()
        {
            base.EndProcessing();
            ExecuteCommand();
        }
        #endregion
        #region Private Methods
        /// <summary>
        /// This is where the actual work is done. It uses WMI to create Win32_Process on the Target server and asynchronously invokes the supplied command. There is no exception handling to ensure that exceptions bubble up to the shell as no remedial action can take place here.
        /// </summary>
        private void ExecuteCommand()
        {
            ConnectionOptions connection = new ConnectionOptions();
            // Point of contention: should we allow empty passwords? Currently this is disallowed by the ValidateNotNullOrEmpty attribute.
            if (!string.IsNullOrEmpty(username))
            {
                // By default, unless you specify connection.Authority, it will use NTLM not Kerberos
                connection.Username = username;
                connection.Password = password;
            }
            ManagementScope scope = new ManagementScope(string.Format("\\\\{0}\\root\\CIMV2", targetServerName), connection);
            scope.Connect();
            ManagementClass classInstance = new ManagementClass(scope, new ManagementPath("Win32_Process"), null);
            ManagementBaseObject inParams = classInstance.GetMethodParameters("Create");
            inParams["CommandLine"] = remoteCommand;
            ManagementBaseObject outParams = classInstance.InvokeMethod("Create", inParams, null);
        }
        #endregion
    }
}

Partition, Federate, Aggregate and Synchronise

James.Saull — Wed, 04 Jun 2008 22:01:02 GMT

It is common practice to use a partitioning function to scale out a solution. For example, you may divide a database among four equal geographical regions and have four separate installations of the database each with the same database but only a quarter of the data. When these four databases outgrew the single physical infrastructure they could be split out on to separate physical machines with a region on each. When throughput of each machine was in jeopardy you could change the partitioning scheme to be more granular - e.g. sub regions. With 40 sub regions you could house 4 sub regions per physical server and have 10 physical servers. And so on and so on. Clearly the application tier needs to be aware of the partitioning scheme and be able to resolve the request it is dealing with to the server that is able to process it. DNS is an example of this resolution by resolving the partition to the handling resource. E.g. www.conchango.com resolves to an IP address which directs your resource request to the appropriate resources.

Remember the partitioning scheme should take in to account how the data is used and not just logical groupings. Ideally your architecture will include the ability to easily re-partition.You may be motivated by reducing the sheer data volumes - solved by creating many smaller databases devoted to a much more manageable data partition. You may be motivated more by sheer TPS (Transactions Per Second) with a perfectly reasonable data volume - solved perhaps by load balancing across identical servers. Ease of re-partitioning allows you to identify hot areas of data attracting activity and break that into many partitions so that you can give each server equal amounts of hot and cold partitions. These hot/cold areas may not be fixed for all time...

Anyway my point is not about comparing all the possible schemes! Get to the point!

Either way you slice it you introduce two key problems...

Aggregate. You are likely to have queries that span partitions. Typically this would be a reporting requirement but sometimes it can be part of the OLTP solution. Spanning partitions might mean different tables, different databases, different servers, different continents... ETL/Data Warehousing techniques have solved this problem.

Synchronising. You might struggle to partition the data because queries need to span the entire dataset; therefore you may create four identical copies of the database and put a load balancer across the top (another form of partition and resolve). But what if the data is Read/Write? Then you must use a replication scheme for the updates so write-reads are consistent even when each part is served by different servers. This will give the problems of replication latency (i.e. you make an update on server 1, but your subsequent read happens against server 2 too soon and the update you made appears to have been lost because it has not arrived on server 2 yet). To avoid replication latency you may use a distributed transaction but this is not ideal as it will defeat scalability. Synchronising can also lead to merge conflicts... There are a variety of solutions but they are dependent on the nature of the problem.

Just thought I'd mention it, because although a lot of people partition and synchronise they forget about the aggregation issue.

People and Process over Technology

James.Saull — Tue, 20 May 2008 20:42:54 GMT

As technology and engineering zealots there is nothing we can't build. Seriously. It is all a question of time, resource and budget. And there's the rub. There comes a point in almost any business process where we hit those points of diminishing returns with technology. All those edge cases where the coincidence of a series of unfortunate events and utterly rare occurrences all align to undermine the beautiful solution. Sometimes the solution is nigh on impossible, or the number of permutations and outcomes to cater for so magnificent. The huge amount of effort required to analyse, test, develop, document, deploy and maintain just can't possibly be justified.

That is not a defeat. That's just economics. I often see this happen especially during business continuity and disaster recovery conversations. The doomsday scenarios are imagined but the solution's Recovery Time Objective remains 1 micro second and the Recovery Point Objective is absolutely no loss of data. When people calm down and realise that such a solution would require the bending of known laws of the universe and cost a significant fortune people start to become more reasonable. How about the call centre people answering the phone use paper and pencil should the system be catastrophically lost. Would that be reasonable? Allow the system 1 hour to be restored into service and have people input the paper-persisted records offline. That might work out a whole lot better!

The other one I often see is a straight through processing system with enormously convoluted compensating transactions and business rules to try and cover everything that could possibly happen with an application. Would it be OK to put the order into an exception bucket where people make judgement calls? Ring the customer back and confirm some details? The emphasis on this solution would be to make it re-entrant and idempotent in all the right places but it could cope with a great deal of edge cases - especially the ones you don't know about or could not imagine.

The important point is to know when to transition your thinking from pure-play technology to a "people and process" solution. In fact you should almost certainly factor this into any solution because you can't anticipate everything. Once the system is in production and you notice certain patterns developing in the "people and process", you can design a solution and put it on the work backlog alongside the demonstrable business case.

People and Process. Factor it in. Then factor it out. Iteratively. If you see what I mean.

History is our greatest teacher

James.Saull — Thu, 24 Apr 2008 13:05:49 GMT

IT is old enough now to see itself repeating itself. It is also old enough for a lot of us not to remember a lot of it and benefit from the history lessons. "Plus ça change, plus c'est la même chose".

One particular example that made me smile was VMWare talking about Lifecycle Manager the ability to use virtualisation to create resources and monitor them and allow you to assign cost metrics. This has sort of thing has been done before and you can see the mainframe guys just nodding sagely.

Sometimes it is the just the economics of the time that make certain techniques more or less relevant. We exploded into distributed computing and despite the total immaturity of the platform it thrived because of the economics. In a lot of enterprises they are trying to rebuild/reinvent the mainframe again; just not on rare proprietary hardware and software.

It'd be nice to have the time to explore more history of I.T. to find inspiration to tackle modern problems. For example, I am sure those mainframe folk will know all the complexities and difficulties of centralised resources (disk, network, CPU etc.) being cross charged to different departments by usage (peak and off peak etc.).

Many of Leonardo da Vinci's designs could not be realised during his time because the materials or manufacturing had not advanced enough. We can apply some new technologies to the old ambitions. Once upon a time centralised mainframes were great, but it was troubled by being such a vast expense the entire company would have to be on board to justify its cost. Today, the economics of hardware, software, datacentre space and management are different. Today we can build dedicated grids or ones that scavenge idle desktop CPU cycles. We can build very large servers for virtualisation or lots of little ones to allow for more autonomy of the business units. We can have DAS, NAS or SAN. We are even seeing more and more "cloud" technologies like Amazon's S3 or EC2 creeping into the equation.

So what problems did they have in the past that feel similar to the ones we have today? How did they solve them, and how would modern evolutions topple the equations toward a much more favourable result today?

Abstractions and Facades vs Fundamentals

James.Saull — Mon, 21 Apr 2008 20:54:48 GMT

Abstractions are a beautiful thing, and it is said that any problem in technology can be solved with another level of abstraction or indirection. Tremendous. But like with most things, there is a flip side.

Once upon a time you would program by hardwiring hardware. This evolved to general purpose hardware and machine code. Higher level abstractions came along with assembler; then C with compilers and so on and so forth. Now we regularly use things like Visual Studio, the Dynamic Language Runtime and Iron Python. Other technologies take the same trajectory. One moment you are writing angle brackets onto a HTTP stream to do SOAP and then next you are adding a service reference to your C# project and you never see an angle bracket ever again because even the XML configuration file has a GUI editor!

All the abstractions are clearly a great thing but the flip side is when things go wrong. If you are relying on SQL Server, for example, and you are not seeing the performance you expect you might employ some basic knowledge of the abstraction by profiling and having SQL suggest indexes. What next? Peel away some of the abstraction. Maybe you need to know about query plans and statistics or files and filegroups and disk configurations. This is what specialists are for. Subject Matter Experts. People who invest in knowing the abstractions and a significant number of layers beneath.

However, there are fundamentals, and whatever the abstractions are we should know these fundamentals (even if some of those are abstractions in their own right!). Each domain will have different fundamentals. What would you choose as your fundamentals? TCPIP, DNS, SMB? XSD, XML, WSDL, SOAP, WSI? HTTP, FTP? Two phase commit? Distinguish between throughput and latency, processes and threads? The list actually gets pretty big. Some fundamentals are just engineering concepts: parallel versus serial or centralised versus distributed etc.

Why am I writing this post? With the explosion of technologies, frameworks and versions it would seem a lot of people are having to trade a knowledge of fundamentals for a vast but cursory knowledge of very high level abstractions and I feel it is hurting. Is it a sign of my age? Is it just that today's abstraction are tomorrow's fundamentals? After all, if I think my vague understanding IL is clever, it is still not machine code, fetch execute cycles and binary (I can't even remember what two's compliment is!) but you have to draw your own boundary at some point!

Fundamentals allow you to decompose your solution and see it for what it really is. It won't matter how many shiny bits it has, it is just the composition of fundamentals. For example, you'll be able to use reflector and see that the code is a single threaded foreach loop making cross machine calls. There is no magic in that - the solution is clear.

I am feeling it more and more lately that great teams are a combination of the generalist and the specialist. They both know their fundamentals and engineering concepts - this can never be lost, but one invests in knowing a very broad range of abstractions and one delves deep. But either way both should recognise what fundamentals are relevant to their domain and keep investing in them too.

I think one of the hardest abstractions I have ever worked with was VB and MTS. It was very productive to crank out COM components in VB. But when things started to go a bit wrong, you were so far from the next level of abstraction you were in a world of pain! Remind me: what is a Single\Multi Threaded Apartment? What is Activation? What is Thread Local Storage and Thread Affinity? I didn't know I just wanted to write my VB!

Nothing feels quite like going live

James.Saull — Fri, 14 Mar 2008 10:01:04 GMT

Today the Queen is to open Heathrow Terminal 5. It seems strange that Terminal 5 is finally opening. I worked for BAA for a large chunk of 2003 when everyone talked about the very distant go live in March 2008. My project was incrementally delivered during 2003, so my stuff went into service a long time ago; but finally 2008 is here!

Divert two rivers, extend the Piccadilly line, extend the Heathrow Express, 5 floors each the size of ten football pitches etc. I found it interesting how various construction exercises were rehearsed elsewhere because when you are spending £4.3bn you really don't need a delay with so many interdependencies - that is millions every day being impacted. Then there were all the other subtle constraints; like doing Europe's largest ever construction program inside the busiest international airport. For a start that means you have a radar ceiling - don't go putting up high cranes and confusing some critical systems! Find a different way.

Whilst on the project, with my excellent panoramic view of the airport, I watched the final Concorde flights come in. The evening flights to New York used to remind me that it was time to head off home!

So, today, I am having a personal little high seeing this achievement climax. I'll bet there are loads of people out there who contributed their myriad skills in many ways and celebrating too. Well done whoever you are and whatever you did.

What did I do? In retrospect, it was a form of Master Data Management for all the assets that go into making Terminal 5, from the HVAC to door furniture. It was all built using SQL Server, ASP .Net and Crystal Reports. Like T5 itself we applied some very rigorous quality to our solution from a very detailed NAnt build including FxCop, NUnit, NDoc, NCover, (NMock was too immature) and XCopy deploy/undeploy - I never even met the operations team that took our documentation and deployment assets - we just handed it over and it worked! We were an agile 2 pizza team with the primary stakeholder sitting at the same desks seeing the solution from the continuous builds. Wasn't much call for Sprint reviews!

Putting the Swiss Army Knife into SSIS

James.Saull — Thu, 07 Feb 2008 11:34:00 GMT

I heard recently that developers imprint on their first ever programming language. My actual computing career started with the Korn shell on AIX UNIX quickly followed by PERL and Java a while later. Maybe this is why I am constantly enthralled to have finally got KSH, PERL and Java all in one: PowerShell. When you read The UNIX Programming Environment by Kernighan and Pike, everything beautifully falls into place - so much elegance. Anyway, its the way I feel about PowerShell too.

One particular e-commerce solution we are building using Microsoft Commerce Server 2007 has a sizeable ETL sitting behind it. It ingests nearly 100 million rows of data (which will grow considerably in future phases) to compose a catalogue with many millions of products and variants. PowerShell is used to manage the transport/protocols in ensuring the multiple data feeds are ready for the ETL process. Microsoft SQL Server Integration Services then manages the process of preparing this data for importing into Commerce Server 2007, for which we have custom SSIS tasks.

It is one thing to have PowerShell running on the outside of SSIS, or even "shelled out to"; but how about being able to actually embed and write PowerShell directly inside SSIS? You'd want it to share the variables, write to the SSIS logger and be aware of executing in the context of SSIS. My colleague Richard Case has done exactly this, please browse here for his blog post and the custom SSIS PowerShell task. Give him some feedback.