James Saull's Blog

At the time, did you know it was the last time?

2008-06-16T21:50:20Z

As a parent you watch your kids go through phases, just you aren't always aware that you are entering a phase or leaving one. For example, when is the last time you actually change a nappy? When was the last time you used the buggy to cart them around? You probably don't realise it at the time. One day you do it not realising it, and then the next day you forget to take the buggy out, then you realise that you didn't really need it and then you don't bother... unwittingly that day was the last day.

When was the last moment I wrote a line of Korn Shell, PERL, Java or VB6? Once upon a time those languages dominated my daily life, and then one day, unknowingly, it was the last day. When will it be the last day that I actually turn a line of C#? Will it be because my colleagues can't take it anymore? Or will it be because, by chance, I find myself on an engagement that forces me to use some special functional programming language to solve a massively parallel computing task and then I never end up with the opportunity to use C# again? Who knows. I wonder what that line of C# will look like? Probably just a closing brace and I won't even know it.

NUDA

2008-06-04T22:43:54Z

With plenty of people anticipating the massively multiple core CPU era, NUMA architectures in regular workstations, and all the wonderful developments in parallel extensions to development languages and software transactional memory to take advantage of them; I see one big bottleneck that looks set to defeat most of this on regular desktop computing: disk IO. My laptop is dual core and is always disk bound. Is disk going to keep up? How long before I can reasonably (just like dual core is very reasonable in a laptop) have multiple large solid state disks in my laptop?

Will they develop NUDA? Non Uniform Disk Architecture? In other words a disk subsystem that I don't have to micro manage that dovetails into NUMA. Once memory is exhausted the OS has to go to pagefile, but that page file should be held on striped solid state disks, and when that is exhausted it should be on striped hard disks and eventually very large storage on non-striped large disks. Or perhaps more like a continuation of NUMA where it would not stripe the disks but instead devote solid state disk volumes to each memory bank so that it's page file is closest and not competed for. That might not work though: if a thread and memory got migrated to a different set of cores and memory it would have to migrate disk data too which would be very bad. Better to stick with shared storage and striping.

Ramble Ramble.

Invoke-RemoteCommand - PowerShell CmdLet to tide me over until PowerShell V2

2008-06-04T22:22:02Z

To tide me over until PowerShell V2 ships and we can perform remote PowerShell out of the box, I have created a simple WMI based CmdLet that will asynchronously execute a command - remotely. You might find it useful too.

The main limitation is that it does not track the result of the remote execution. You should be able to call any .EXE such as "cmd.exe /c" or "powershell.exe -command".

Here is the C# snippet - I don't code much so use with caution!

using System;
using System.Collections;
using System.Collections.Generic;
using System.ComponentModel;
using System.Diagnostics;
using System.IO;
using System.Management;
using System.Management.Automation;
using Microsoft.PowerShell.Commands;

namespace JamesSaull.PowerShell
{
    /// <summary>
    /// InvokeRemoteCommand is a PowerShell CmdLet using WMI to execute a command on a remote server.
    /// </summary>
    /// <example>Invoke-RemoteCommand -TargetServerName WEB01 -RemoteCommand "cmd.exe /c mkdir c:\temp\testfolder"</example>
    [Cmdlet("Invoke", "RemoteCommand")]
    public class InvokeRemoteCommand : DCPCommandLet
    {
        #region Fields
        private string remoteCommand;
        private string targetServerName;
        private string username = string.Empty;
        private string password = string.Empty;
        #endregion
        #region Command Arguments / Public Properties
        /// <summary>
        /// Name of the server where the command should be executed.
        /// </summary>
        [Parameter(Position = 0, HelpMessage = "Name of the server where the command should be executed.", Mandatory = true)]
        [ValidateNotNullOrEmpty]
        public string TargetServerName
        {
            get { return targetServerName; }
            set { targetServerName = value; }
        }
        /// <summary>
        /// The command to be executed. e.g. cmd.exe /c mkdir c:\temp\testfolder
        /// </summary>
        [Parameter(Position = 1, HelpMessage = "The command to be executed. e.g. cmd.exe /c mkdir c:\temp\testfolder", Mandatory = true)]
        [ValidateNotNullOrEmpty]
        public string RemoteCommand
        {
            get { return remoteCommand; }
            set { remoteCommand = value; }
        }
        /// <summary>
        /// Username used to authenticate and authorise against Target Server. Not allowed for local connections.
        /// </summary>
        [Parameter(Position = 3, HelpMessage = "Username used to authenticate and authorise against Target Server. Not allowed for local connections.")]
        [ValidateNotNullOrEmpty]
        public string Username
        {
            get { return username; }
            set { username = value; }
        }
        /// <summary>
        /// Password used to authenticate and authorise against Target Server.
        /// </summary>
        [Parameter(Position = 4, HelpMessage = "Password used to authenticate and authorise against Target Server.")]
        [ValidateNotNullOrEmpty]
        public string Password
        {
            get { return password; }
            set { password = value; }
        }
        #endregion
        #region Overriding CmdLet
        /// <summary>
        /// Currently this CmdLet does not allow usage in a pipeline so it only overrides EndProcessing and not ProcessRequest or BeginProcessing.
        /// </summary>
        protected override void EndProcessing()
        {
            base.EndProcessing();
            ExecuteCommand();
        }
        #endregion
        #region Private Methods
        /// <summary>
        /// This is where the actual work is done. It uses WMI to create Win32_Process on the Target server and asynchronously invokes the supplied command. There is no exception handling to ensure that exceptions bubble up to the shell as no remedial action can take place here.
        /// </summary>
        private void ExecuteCommand()
        {
            ConnectionOptions connection = new ConnectionOptions();
            // Point of contention: should we allow empty passwords? Currently this is disallowed by the ValidateNotNullOrEmpty attribute.
            if (!string.IsNullOrEmpty(username))
            {
                // By default, unless you specify connection.Authority, it will use NTLM not Kerberos
                connection.Username = username;
                connection.Password = password;
            }
            ManagementScope scope = new ManagementScope(string.Format("\\\\{0}\\root\\CIMV2", targetServerName), connection);
            scope.Connect();
            ManagementClass classInstance = new ManagementClass(scope, new ManagementPath("Win32_Process"), null);
            ManagementBaseObject inParams = classInstance.GetMethodParameters("Create");
            inParams["CommandLine"] = remoteCommand;
            ManagementBaseObject outParams = classInstance.InvokeMethod("Create", inParams, null);
        }
        #endregion
    }
}

Partition, Federate, Aggregate and Synchronise

2008-06-04T22:01:02Z

It is common practice to use a partitioning function to scale out a solution. For example, you may divide a database among four equal geographical regions and have four separate installations of the database each with the same database but only a quarter of the data. When these four databases outgrew the single physical infrastructure they could be split out on to separate physical machines with a region on each. When throughput of each machine was in jeopardy you could change the partitioning scheme to be more granular - e.g. sub regions. With 40 sub regions you could house 4 sub regions per physical server and have 10 physical servers. And so on and so on. Clearly the application tier needs to be aware of the partitioning scheme and be able to resolve the request it is dealing with to the server that is able to process it. DNS is an example of this resolution by resolving the partition to the handling resource. E.g. www.conchango.com resolves to an IP address which directs your resource request to the appropriate resources.

Remember the partitioning scheme should take in to account how the data is used and not just logical groupings. Ideally your architecture will include the ability to easily re-partition.You may be motivated by reducing the sheer data volumes - solved by creating many smaller databases devoted to a much more manageable data partition. You may be motivated more by sheer TPS (Transactions Per Second) with a perfectly reasonable data volume - solved perhaps by load balancing across identical servers. Ease of re-partitioning allows you to identify hot areas of data attracting activity and break that into many partitions so that you can give each server equal amounts of hot and cold partitions. These hot/cold areas may not be fixed for all time...

Anyway my point is not about comparing all the possible schemes! Get to the point!

Either way you slice it you introduce two key problems...

Aggregate. You are likely to have queries that span partitions. Typically this would be a reporting requirement but sometimes it can be part of the OLTP solution. Spanning partitions might mean different tables, different databases, different servers, different continents... ETL/Data Warehousing techniques have solved this problem.

Synchronising. You might struggle to partition the data because queries need to span the entire dataset; therefore you may create four identical copies of the database and put a load balancer across the top (another form of partition and resolve). But what if the data is Read/Write? Then you must use a replication scheme for the updates so write-reads are consistent even when each part is served by different servers. This will give the problems of replication latency (i.e. you make an update on server 1, but your subsequent read happens against server 2 too soon and the update you made appears to have been lost because it has not arrived on server 2 yet). To avoid replication latency you may use a distributed transaction but this is not ideal as it will defeat scalability. Synchronising can also lead to merge conflicts... There are a variety of solutions but they are dependent on the nature of the problem.

Just thought I'd mention it, because although a lot of people partition and synchronise they forget about the aggregation issue.

People and Process over Technology

2008-05-20T20:42:54Z

As technology and engineering zealots there is nothing we can't build. Seriously. It is all a question of time, resource and budget. And there's the rub. There comes a point in almost any business process where we hit those points of diminishing returns with technology. All those edge cases where the coincidence of a series of unfortunate events and utterly rare occurrences all align to undermine the beautiful solution. Sometimes the solution is nigh on impossible, or the number of permutations and outcomes to cater for so magnificent. The huge amount of effort required to analyse, test, develop, document, deploy and maintain just can't possibly be justified.

That is not a defeat. That's just economics. I often see this happen especially during business continuity and disaster recovery conversations. The doomsday scenarios are imagined but the solution's Recovery Time Objective remains 1 micro second and the Recovery Point Objective is absolutely no loss of data. When people calm down and realise that such a solution would require the bending of known laws of the universe and cost a significant fortune people start to become more reasonable. How about the call centre people answering the phone use paper and pencil should the system be catastrophically lost. Would that be reasonable? Allow the system 1 hour to be restored into service and have people input the paper-persisted records offline. That might work out a whole lot better!

The other one I often see is a straight through processing system with enormously convoluted compensating transactions and business rules to try and cover everything that could possibly happen with an application. Would it be OK to put the order into an exception bucket where people make judgement calls? Ring the customer back and confirm some details? The emphasis on this solution would be to make it re-entrant and idempotent in all the right places but it could cope with a great deal of edge cases - especially the ones you don't know about or could not imagine.

The important point is to know when to transition your thinking from pure-play technology to a "people and process" solution. In fact you should almost certainly factor this into any solution because you can't anticipate everything. Once the system is in production and you notice certain patterns developing in the "people and process", you can design a solution and put it on the work backlog alongside the demonstrable business case.

People and Process. Factor it in. Then factor it out. Iteratively. If you see what I mean.

History is our greatest teacher

2008-04-24T13:05:49Z

IT is old enough now to see itself repeating itself. It is also old enough for a lot of us not to remember a lot of it and benefit from the history lessons. "Plus ça change, plus c'est la même chose".

One particular example that made me smile was VMWare talking about Lifecycle Manager the ability to use virtualisation to create resources and monitor them and allow you to assign cost metrics. This has sort of thing has been done before and you can see the mainframe guys just nodding sagely.

Sometimes it is the just the economics of the time that make certain techniques more or less relevant. We exploded into distributed computing and despite the total immaturity of the platform it thrived because of the economics. In a lot of enterprises they are trying to rebuild/reinvent the mainframe again; just not on rare proprietary hardware and software.

It'd be nice to have the time to explore more history of I.T. to find inspiration to tackle modern problems. For example, I am sure those mainframe folk will know all the complexities and difficulties of centralised resources (disk, network, CPU etc.) being cross charged to different departments by usage (peak and off peak etc.).

Many of Leonardo da Vinci's designs could not be realised during his time because the materials or manufacturing had not advanced enough. We can apply some new technologies to the old ambitions. Once upon a time centralised mainframes were great, but it was troubled by being such a vast expense the entire company would have to be on board to justify its cost. Today, the economics of hardware, software, datacentre space and management are different. Today we can build dedicated grids or ones that scavenge idle desktop CPU cycles. We can build very large servers for virtualisation or lots of little ones to allow for more autonomy of the business units. We can have DAS, NAS or SAN. We are even seeing more and more "cloud" technologies like Amazon's S3 or EC2 creeping into the equation.

So what problems did they have in the past that feel similar to the ones we have today? How did they solve them, and how would modern evolutions topple the equations toward a much more favourable result today?

Abstractions and Facades vs Fundamentals

2008-04-21T20:54:48Z

Abstractions are a beautiful thing, and it is said that any problem in technology can be solved with another level of abstraction or indirection. Tremendous. But like with most things, there is a flip side.

Once upon a time you would program by hardwiring hardware. This evolved to general purpose hardware and machine code. Higher level abstractions came along with assembler; then C with compilers and so on and so forth. Now we regularly use things like Visual Studio, the Dynamic Language Runtime and Iron Python. Other technologies take the same trajectory. One moment you are writing angle brackets onto a HTTP stream to do SOAP and then next you are adding a service reference to your C# project and you never see an angle bracket ever again because even the XML configuration file has a GUI editor!

All the abstractions are clearly a great thing but the flip side is when things go wrong. If you are relying on SQL Server, for example, and you are not seeing the performance you expect you might employ some basic knowledge of the abstraction by profiling and having SQL suggest indexes. What next? Peel away some of the abstraction. Maybe you need to know about query plans and statistics or files and filegroups and disk configurations. This is what specialists are for. Subject Matter Experts. People who invest in knowing the abstractions and a significant number of layers beneath.

However, there are fundamentals, and whatever the abstractions are we should know these fundamentals (even if some of those are abstractions in their own right!). Each domain will have different fundamentals. What would you choose as your fundamentals? TCPIP, DNS, SMB? XSD, XML, WSDL, SOAP, WSI? HTTP, FTP? Two phase commit? Distinguish between throughput and latency, processes and threads? The list actually gets pretty big. Some fundamentals are just engineering concepts: parallel versus serial or centralised versus distributed etc.

Why am I writing this post? With the explosion of technologies, frameworks and versions it would seem a lot of people are having to trade a knowledge of fundamentals for a vast but cursory knowledge of very high level abstractions and I feel it is hurting. Is it a sign of my age? Is it just that today's abstraction are tomorrow's fundamentals? After all, if I think my vague understanding IL is clever, it is still not machine code, fetch execute cycles and binary (I can't even remember what two's compliment is!) but you have to draw your own boundary at some point!

Fundamentals allow you to decompose your solution and see it for what it really is. It won't matter how many shiny bits it has, it is just the composition of fundamentals. For example, you'll be able to use reflector and see that the code is a single threaded foreach loop making cross machine calls. There is no magic in that - the solution is clear.

I am feeling it more and more lately that great teams are a combination of the generalist and the specialist. They both know their fundamentals and engineering concepts - this can never be lost, but one invests in knowing a very broad range of abstractions and one delves deep. But either way both should recognise what fundamentals are relevant to their domain and keep investing in them too.

I think one of the hardest abstractions I have ever worked with was VB and MTS. It was very productive to crank out COM components in VB. But when things started to go a bit wrong, you were so far from the next level of abstraction you were in a world of pain! Remind me: what is a Single\Multi Threaded Apartment? What is Activation? What is Thread Local Storage and Thread Affinity? I didn't know I just wanted to write my VB!

Nothing feels quite like going live

2008-03-14T10:01:04Z

Today the Queen is to open Heathrow Terminal 5. It seems strange that Terminal 5 is finally opening. I worked for BAA for a large chunk of 2003 when everyone talked about the very distant go live in March 2008. My project was incrementally delivered during 2003, so my stuff went into service a long time ago; but finally 2008 is here!

Divert two rivers, extend the Piccadilly line, extend the Heathrow Express, 5 floors each the size of ten football pitches etc. I found it interesting how various construction exercises were rehearsed elsewhere because when you are spending £4.3bn you really don't need a delay with so many interdependencies - that is millions every day being impacted. Then there were all the other subtle constraints; like doing Europe's largest ever construction program inside the busiest international airport. For a start that means you have a radar ceiling - don't go putting up high cranes and confusing some critical systems! Find a different way.

Whilst on the project, with my excellent panoramic view of the airport, I watched the final Concorde flights come in. The evening flights to New York used to remind me that it was time to head off home!

So, today, I am having a personal little high seeing this achievement climax. I'll bet there are loads of people out there who contributed their myriad skills in many ways and celebrating too. Well done whoever you are and whatever you did.

What did I do? In retrospect, it was a form of Master Data Management for all the assets that go into making Terminal 5, from the HVAC to door furniture. It was all built using SQL Server, ASP .Net and Crystal Reports. Like T5 itself we applied some very rigorous quality to our solution from a very detailed NAnt build including FxCop, NUnit, NDoc, NCover, (NMock was too immature) and XCopy deploy/undeploy - I never even met the operations team that took our documentation and deployment assets - we just handed it over and it worked! We were an agile 2 pizza team with the primary stakeholder sitting at the same desks seeing the solution from the continuous builds. Wasn't much call for Sprint reviews!

Putting the Swiss Army Knife into SSIS

2008-02-07T11:34:00Z

I heard recently that developers imprint on their first ever programming language. My actual computing career started with the Korn shell on AIX UNIX quickly followed by PERL and Java a while later. Maybe this is why I am constantly enthralled to have finally got KSH, PERL and Java all in one: PowerShell. When you read The UNIX Programming Environment by Kernighan and Pike, everything beautifully falls into place - so much elegance. Anyway, its the way I feel about PowerShell too.

One particular e-commerce solution we are building using Microsoft Commerce Server 2007 has a sizeable ETL sitting behind it. It ingests nearly 100 million rows of data (which will grow considerably in future phases) to compose a catalogue with many millions of products and variants. PowerShell is used to manage the transport/protocols in ensuring the multiple data feeds are ready for the ETL process. Microsoft SQL Server Integration Services then manages the process of preparing this data for importing into Commerce Server 2007, for which we have custom SSIS tasks.

It is one thing to have PowerShell running on the outside of SSIS, or even "shelled out to"; but how about being able to actually embed and write PowerShell directly inside SSIS? You'd want it to share the variables, write to the SSIS logger and be aware of executing in the context of SSIS. My colleague Richard Case has done exactly this, please browse here for his blog post and the custom SSIS PowerShell task. Give him some feedback.

Quantum leaps and more performant systems

2007-10-31T20:53:49Z

I am sure I am not supposed to whinge when blogging but this itch needs scratching.

Far too often I hear, typically pedantic, technical folk explain the latest and greatest technology in the following way: "... and the wonder-widget can lead to a quantum leap in performance and scalability...". Do they really mean to say that the "wonder-widget" gives rise to the smallest measurable improvement in performance and scalability? Given the context of the pitch and fervour of the orator, I doubt it. How, when used as an adjective, did this come to mean a "significant" amount? Quantum mechanics is all about the small stuff.

The other one that really grates is "performant". Tweaking the threading and fiddling the concurrency will deliver the most performant system. It may deliver a high-performance system, or a highly-efficient system, or maybe even a well-performing system - just not a performant system. Performant means "a performer" as in "an actor".

Somehow these uses have forced themselves into mainstream use and the language will adapt and adopt them. Grrrr.

PowerShell is the new Swiss Army Knife

2007-09-21T09:10:29Z

Recently I have been working through some thorny data and encoding issues whilst integrating via BizTalk. PowerShell really is the new Swiss Army Knife that Perl always was in my long past days of Korn shell on AIX UNIX. As an example I have had to strip out a particular field from a c. 1GB XML file and get a unique count. Typically on a 32bit system, 1GB won't load into a DOM to just do XmlDocument.SelectNodes("...").Count. So I wrote a few lines of C# in a ConsoleApplication that used a XmlTextReader to just Console.WriteLine the element value every time it encountered a particular element. I can reuse this code over and over. So here is a bit of PowerShell to manipulate the output:

get-content C:\temp\mylist.txt | sort -caseSensitive | where {$_.Length -gt 0} | get-unique

This gives me a unique list whilst skipping empty lines. I could go further just in case there is some whitespace crawling in:

get-content C:\temp\mylist.txt | %{$_.Trim()} | where {$_.Length -gt 0} | sort -caseSensitive | get-unique

Maybe I want that counted:

get-content C:\temp\mylist.txt | %{$_.Trim()} | where {$_.Length -gt 0} | sort -caseSensitive | get-unique| measure-object

Another issue we faced was that we receiving files using different encodings. What tool should I use to check to view the Hex and see what the bytes in the stream really say and check the difference? PowerShell:

3> gc C:\temp\FileOne.xml -encoding byte -totalcount 1000 | format-hex

Address: 0 1 2 3 4 5 6 7 8 9 A B C D E F 10 11 12 13 14 15 16 17 ASCII

-------- ----------------------------------------------------------------------- ------------------------

00000000 EF BB BF 3C 44 69 73 70 6C 61 79 4E 61 6D 65 20 56 61 6C 75 65 3D 22 43 ...<DisplayName Value="C

00000018 65 73 C3 A1 72 69 61 20 C3 89 76 6F 72 61 22 20 6C 61 6E 67 75 61 67 65 es..ria ..vora" language

00000030 3D 22 65 6E 2D 47 42 22 20 2F 3E ="en-GB" />

Here I can see that the UTF-8 Byte Order Mark is present and what bytes are being used to represent the accented characters in the above XML.

To someone who started out as a UNIX sysadmin / Oracle DBA you have no idea what it is like to have a proper shell again!

Genchi Genbutsu Architects

2007-08-17T19:18:09Z

As a technical architect at Conchango my remit of technologies has become much broader than ever before. With breadth comes two problems: so wide, but so shallow as to add no value or you work yourself to death with the unrealistic goal of knowing everything about everything.

It is a common complaint amongst the technical community that the field has become so massive that it is no longer realistic to know all about all. There is a finite amount of time to absorb and experience. Period. So what gives?

My role requires that I know a fair amount about .Net (including ASP.Net, WF, WCF, Silverlight, MSBuild, Sandcastle etc.), SQL Server (including mirroring, log shipping, service broker, SSIS, Reporting Services etc.), BizTalk Server, MOSS, FAST, Scene 7, CoreMetrics, WebTrends, Commerce Server 2007, Content Delivery Networks, Performance, Scalability, Availability, Security, Maintainability, Operations, Hosting, Supportability. I have to understand how these products are sized, licensed and deployed. It is also good for me to be aware of other solutions such as DotNetNuke, Astoria, PopFly or relevant open source projects such as the Community Kit for SharePoint. Then there are all the upcoming versions, roadmaps and visions. It is just endless and I know I am not alone facing this challenge.

So how do I do this but remain relevant? I read as much as I can (MSDN, RSS and wherever the links take me). I listen to as many podcasts as I can. Of course I actually work with these technologies on a day to day basis. That's still not enough.

Conchango is loaded with Subject Matter Experts (SME) that are totally brilliant in all sorts of wonderful ways from Business Intelligence to Silverlight, from WebLogic to Ruby, from ISEB practitioners to Agile leading lights, from User Experience and design experts to Retail gurus. The only way I can possibly do what I do is to identify with each of the SMEs with whom I can relate to a sufficient degree. For example, when helping to put an architecture together I don't pretend I know how the backplane of a SAN is spread across each of the trays in a HP EVA SAN and where there might be single points of failure. I know someone who does. I can explain to him the architecturally significant challenges that I face and I can learn and leverage his expertise and put it into the context I need. This is what is so awesome about working with so many gifted people with so much passion. I can cover so much more when standing on the shoulders of giants.

There is still one problem. Genchi Genbutsu. Getting involved and remaining in contact with the delivery of customer solutions is a vital way to keeping it real and making sure one doesn't become a "Marchitect". This is where the marketing misleads the lazy architect about the reality of solutions. I have encountered marchitects far too often and it is embarrassing. They represent the pinnacle of technology yet they are so clearly out of touch with reality and doom so much and so many to certain failure. How many times have you seen the "ivory tower" architecture departments? I hope someone tells me if I stray into this camp.

Genchi Genbutsu is just vital.

Service availability is the product of its parts

2007-04-26T14:01:06Z

When something crops up a few times I feel like making a post. It is important to remember that when evaluating an architecture, especially on its "availability" credentials it is important to consider the service and everything within its boundary: all its layers, subsystems, dependencies and so forth.

An architecture that comprises one component with shared nothing (e.g. a web server with static content) can be made very highly available very cost effectively through scale out redundancy. This is a very wide and very shallow architecture. The same unit of deployment can be replicated around the globe on different networks etc.

An architecture that is very narrow and very deep is likely to struggle to meet the same degree of availability. If the web server depends on a database and other web services with a series of dependencies beneath it then it becomes very hard. With the web servers dependent on the database there is a single point of failure that could defeat the scale out redundancy of the web tier, so you employ database clustering, but how do you distribute the database for geo-clustering... The cost goes up.

Despite the cost issue, to purely evaluate the availability of the narrow and deep architecture you need to consider each component. If the web service tier is built and operated to 99.9% availability, and the database to 99.9% and the other web services to 99.9%. The actual "service" availability is in fact 99.9% x 99.9% x 99.9%. That is over a whole day per annum. To bring the "service" back to 99.9% availability, each component (if shared equally) would actually have to run at about 99.97%. That is equivalent to reducing the downtime of each component from 8 hours 45 minutes per annum to 2 hours 55 minutes. That is a big difference. If the requirement for the service was 99.99% availability, each part is only allowed just under 18 minutes outage per annum. The cost and complexity to deliver ever increasing availability can be very high. Some of the downtime can be planned and of course that the downtime all occurs at the same time, but unplanned downtime is never that conveniently stacked!

Architecting a system that is very wide and very shallow (very redundant and very independent) is easy to achieve availability and scalability with very good economics. A system that struggles to be wide at any layer and is also very deep (i.e. not easily redundant and very dependent) will struggle to achieve high availability and scalability with good economics.

Availability is not always a high priority, or not the only priority, but when it is remember to factor in the whole service not just each layer.

Is a complex architecture wrong?

2007-04-26T13:08:22Z

Sometimes a solution is complex and when confronted by this, many people follow their innate desire to find a more simple and "elegant" solution. This of course is a noble endeavour. A simple and elegant solution is probably less risky to schedule, cost and resource. It is probably far more cost effective to operate and maintain. It is also probably more available, scalable, highly performing, secure and so forth.

The counter point to this is perhaps nicely summarised by Albert Einstein (thanks to colleague Sid Kargupta for pointing this out): "Everything should be made as simple as possible, but not simpler". To architects this reminds us that some solutions are inherently, fundamentally and irreconcilably complex either in part or whole. Attempting to make these parts any simpler will mean that the solution begins to lose something. It is better to recognise and respect the nature of the problem than to undermine it.

To quote someone quite different to Albert Einstein: "Shoot the hostage". If you remember Keanu Reeves in Speed you'll know what I am referring to. The character in the film essentially, albeit unusually, eliminates the hostage situation allowing him to pursue the criminal. In the architect's case, when presented with a solution that cannot be further reduced, it is not a bad strategy to go back to the client to re-examine the requirements that culminate in the complexity. If you can shoot the hostage it might be better all round. Does the customer realise the ramifications of their requirement? Was the requirement born of absolute necessity without compromise or was it a reasonable expectation given the understanding at the time?

A classic case of this is the "the solution must be five nines availability" demand. Some solutions will have a hard time solving this problem full stop. Add the cost of such a solution into the mix and often it is not commensurate with the risk and exposure the requirement is seeking to mitigate. The requirement is reconsidered, giving scope to accommodate different solutions.

So, no, complex architectures are not wrong as long as they are simple as they can be (no simpler) and that the requirements that culminate in the complexity have been thoroughly validated in the new light shed by the implications.

ASP.Net 2.0 Session State Partitioning

2007-04-03T19:29:51Z

It is very common to scale out web applications by deploying a large array of inexpensive commodity servers as a very economical way to achieve high scalability and availability. However, once you have put this into play for the presentation and business logic layers you get to the thorny issue of partitioning the data that drives the web application. This usually requires plenty of thought. What scheme will you use to partition the data? How will you distribute the data? How will you query across the data now that it is partitioned and federated? Actually there is a lot more to think about but that is not why I am posting.

One thing that often slips under most people's radar is web application session state. In a vast farm of web servers, how do you maintain session state when a load balancer could send you to any one of the servers for each request?

Make the application stateless
Maintain state in the browser (e.g. a cookie)
Maintain state in the URL
Maintain server affinity with the client using sticky sessions or redirecting the first ever call to a DNS distinguished server (server129.mydomain.com)
etc.

If none of those work so well in your context you might want to apply the "partition and federate" technique to the session store itself. It is a little known fact that this is supported under ASP.Net 2.0. As an application developer you can obliviously use session state as normal because you have defined your partitioning scheme in code and configured either a SQL Server or State Server farm to scale out session. This is ideal for many scenarios, especially applications that are already built and deployed and don't have the luxury of examining and building their preferred solution from the get-go.

Check it out, it is an option worth remembering:

http://msdn.microsoft.com/msdnmag/issues/05/09/sessionstate/

"ASP.NET 2.0 provides a solution to the problems encountered when scaling up by enabling horizontal scale-out of session state stores through its state partitioning feature. State partitioning enables the session data and the associated processing load to be divided between multiple out-of-process state stores, allowing the session state load to scale as the Web farm grows and the number of concurrent sessions increases. It works by supplying a custom partitioning algorithm to SessionStateModule, which uses the algorithm to determine the state store connection string to be used for the current request based on the session ID. Both the SQLServer and the StateServer providers will then use the appropriate connection string to fetch and save the session..."