blogs.conchango.com

Abstractions are a beautiful thing, and it is said that any problem in technology can be solved with another level of abstraction or indirection. Tremendous. But like with most things, there is a flip side.

Once upon a time you would program by hardwiring hardware. This evolved to general purpose hardware and machine code. Higher level abstractions came along with assembler; then C with compilers and so on and so forth. Now we regularly use things like Visual Studio, the Dynamic Language Runtime and Iron Python. Other technologies take the same trajectory. One moment you are writing angle brackets onto a HTTP stream to do SOAP and then next you are adding a service reference to your C# project and you never see an angle bracket ever again because even the XML configuration file has a GUI editor!

All the abstractions are clearly a great thing but the flip side is when things go wrong. If you are relying on SQL Server, for example, and you are not seeing the performance you expect you might employ some basic knowledge of the abstraction by profiling and having SQL suggest indexes. What next? Peel away some of the abstraction. Maybe you need to know about query plans and statistics or files and filegroups and disk configurations. This is what specialists are for. Subject Matter Experts. People who invest in knowing the abstractions and a significant number of layers beneath.

However, there are fundamentals, and whatever the abstractions are we should know these fundamentals (even if some of those are abstractions in their own right!). Each domain will have different fundamentals. What would you choose as your fundamentals? TCPIP, DNS, SMB? XSD, XML, WSDL, SOAP, WSI? HTTP, FTP? Two phase commit? Distinguish between throughput and latency, processes and threads? The list actually gets pretty big. Some fundamentals are just engineering concepts: parallel versus serial or centralised versus distributed etc.

Why am I writing this post? With the explosion of technologies, frameworks and versions it would seem a lot of people are having to trade a knowledge of fundamentals for a vast but cursory knowledge of very high level abstractions and I feel it is hurting. Is it a sign of my age? Is it just that today's abstraction are tomorrow's fundamentals? After all, if I think my vague understanding IL is clever, it is still not machine code, fetch execute cycles and binary (I can't even remember what two's compliment is!) but you have to draw your own boundary at some point!

Fundamentals allow you to decompose your solution and see it for what it really is. It won't matter how many shiny bits it has, it is just the composition of fundamentals. For example, you'll be able to use reflector and see that the code is a single threaded foreach loop making cross machine calls. There is no magic in that - the solution is clear.

I am feeling it more and more lately that great teams are a combination of the generalist and the specialist. They both know their fundamentals and engineering concepts - this can never be lost, but one invests in knowing a very broad range of abstractions and one delves deep. But either way both should recognise what fundamentals are relevant to their domain and keep investing in them too.

I think one of the hardest abstractions I have ever worked with was VB and MTS. It was very productive to crank out COM components in VB. But when things started to go a bit wrong, you were so far from the next level of abstraction you were in a world of pain! Remind me: what is a Single\Multi Threaded Apartment? What is Activation? What is Thread Local Storage and Thread Affinity? I didn't know I just wanted to write my VB!

I heard recently that developers imprint on their first ever programming language. My actual computing career started with the Korn shell on AIX UNIX quickly followed by PERL and Java a while later. Maybe this is why I am constantly enthralled to have finally got KSH, PERL and Java all in one: PowerShell. When you read The UNIX Programming Environment by Kernighan and Pike, everything beautifully falls into place - so much elegance. Anyway, its the way I feel about PowerShell too.

One particular e-commerce solution we are building using Microsoft Commerce Server 2007 has a sizeable ETL sitting behind it. It ingests nearly 100 million rows of data (which will grow considerably in future phases) to compose a catalogue with many millions of products and variants. PowerShell is used to manage the transport/protocols in ensuring the multiple data feeds are ready for the ETL process. Microsoft SQL Server Integration Services then manages the process of preparing this data for importing into Commerce Server 2007, for which we have custom SSIS tasks.

It is one thing to have PowerShell running on the outside of SSIS, or even "shelled out to"; but how about being able to actually embed and write PowerShell directly inside SSIS? You'd want it to share the variables, write to the SSIS logger and be aware of executing in the context of SSIS. My colleague Richard Case has done exactly this, please browse here for his blog post and the custom SSIS PowerShell task. Give him some feedback.

Recently I have been working through some thorny data and encoding issues whilst integrating via BizTalk. PowerShell really is the new Swiss Army Knife that Perl always was in my long past days of Korn shell on AIX UNIX. As an example I have had to strip out a particular field from a c. 1GB XML file and get a unique count. Typically on a 32bit system, 1GB won't load into a DOM to just do XmlDocument.SelectNodes("...").Count. So I wrote a few lines of C# in a ConsoleApplication that used a XmlTextReader to just Console.WriteLine the element value every time it encountered a particular element. I can reuse this code over and over. So here is a bit of PowerShell to manipulate the output:

get-content C:\temp\mylist.txt | sort -caseSensitive | where {$_.Length -gt 0} | get-unique

This gives me a unique list whilst skipping empty lines. I could go further just in case there is some whitespace crawling in:

get-content C:\temp\mylist.txt | %{$_.Trim()} | where {$_.Length -gt 0} | sort -caseSensitive | get-unique

Maybe I want that counted:

get-content C:\temp\mylist.txt | %{$_.Trim()} | where {$_.Length -gt 0} | sort -caseSensitive | get-unique| measure-object

Another issue we faced was that we receiving files using different encodings. What tool should I use to check to view the Hex and see what the bytes in the stream really say and check the difference? PowerShell:

3> gc C:\temp\FileOne.xml -encoding byte -totalcount 1000 | format-hex

Address:  0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F 10 11 12 13 14 15 16 17 ASCII

-------- ----------------------------------------------------------------------- ------------------------

00000000 EF BB BF 3C 44 69 73 70 6C 61 79 4E 61 6D 65 20 56 61 6C 75 65 3D 22 43 ...<DisplayName Value="C

00000018 65 73 C3 A1 72 69 61 20 C3 89 76 6F 72 61 22 20 6C 61 6E 67 75 61 67 65 es..ria ..vora" language

00000030 3D 22 65 6E 2D 47 42 22 20 2F 3E                                        ="en-GB" />

Here I can see that the UTF-8 Byte Order Mark is present and what bytes are being used to represent the accented characters in the above XML.

To someone who started out as a UNIX sysadmin / Oracle DBA you have no idea what it is like to have a proper shell again!

When something crops up a few times I feel like making a post. It is important to remember that when evaluating an architecture, especially on its "availability" credentials it is important to consider the service and everything within its boundary: all its layers, subsystems, dependencies and so forth.

An architecture that comprises one component with shared nothing (e.g. a web server with static content) can be made very highly available very cost effectively through scale out redundancy. This is a very wide and very shallow architecture. The same unit of deployment can be replicated around the globe on different networks etc.

An architecture that is very narrow and very deep is likely to struggle to meet the same degree of availability. If the web server depends on a database and other web services with a series of dependencies beneath it then it becomes very hard. With the web servers dependent on the database there is a single point of failure that could defeat the scale out redundancy of the web tier, so you employ database clustering, but how do you distribute the database for geo-clustering... The cost goes up.

Despite the cost issue, to purely evaluate the availability of the narrow and deep architecture you need to consider each component. If the web service tier is built and operated to 99.9% availability, and the database to 99.9% and the other web services to 99.9%. The actual "service" availability is in fact 99.9% x 99.9% x 99.9%. That is over a whole day per annum. To bring the "service" back to 99.9% availability, each component (if shared equally) would actually have to run at about 99.97%. That is equivalent to reducing the downtime of each component from 8 hours 45 minutes per annum to 2 hours 55 minutes. That is a big difference. If the requirement for the service was 99.99% availability, each part is only allowed just under 18 minutes outage per annum. The cost and complexity to deliver ever increasing availability can be very high. Some of the downtime can be planned and of course that the downtime all occurs at the same time, but unplanned downtime is never that conveniently stacked!

Architecting a system that is very wide and very shallow (very redundant and very independent) is easy to achieve availability and scalability with very good economics. A system that struggles to be wide at any layer and is also very deep (i.e. not easily redundant and very dependent) will struggle to achieve high availability and scalability with good economics.

Availability is not always a high priority, or not the only priority, but when it is remember to factor in the whole service not just each layer.

It is very common to scale out web applications by deploying a large array of inexpensive commodity servers as a very economical way to achieve high scalability and availability. However, once you have put this into play for the presentation and business logic layers you get to the thorny issue of partitioning the data that drives the web application. This usually requires plenty of thought. What scheme will you use to partition the data? How will you distribute the data? How will you query across the data now that it is partitioned and federated? Actually there is a lot more to think about but that is not why I am posting.

One thing that often slips under most people's radar is web application session state. In a vast farm of web servers, how do you maintain session state when a load balancer could send you to any one of the servers for each request?

• Make the application stateless
• Maintain state in the browser (e.g. a cookie)
• Maintain state in the URL
• Maintain server affinity with the client using sticky sessions or redirecting the first ever call to a DNS distinguished server (server129.mydomain.com)
• etc.

If none of those work so well in your context you might want to apply the "partition and federate" technique to the session store itself. It is a little known fact that this is supported under ASP.Net 2.0. As an application developer you can obliviously use session state as normal because you have defined your partitioning scheme in code and configured either a SQL Server or State Server farm to scale out session. This is ideal for many scenarios, especially applications that are already built and deployed and don't have the luxury of examining and building their preferred solution from the get-go.

Check it out, it is an option worth remembering:

http://msdn.microsoft.com/msdnmag/issues/05/09/sessionstate/

"ASP.NET 2.0 provides a solution to the problems encountered when scaling up by enabling horizontal scale-out of session state stores through its state partitioning feature. State partitioning enables the session data and the associated processing load to be divided between multiple out-of-process state stores, allowing the session state load to scale as the Web farm grows and the number of concurrent sessions increases. It works by supplying a custom partitioning algorithm to SessionStateModule, which uses the algorithm to determine the state store connection string to be used for the current request based on the session ID. Both the SQLServer and the StateServer providers will then use the appropriate connection string to fetch and save the session..."

Back at university I remember starting my thermodynamics course with the laws of thermodynamics. The reason they were useful was that whenever you approached any thermodynamic problem such as axial flow jet engines, these apparently simple laws always governed the ultimate complex mathematical descriptions of these systems. If the application of a law to an equation failed, then the equation was flawed.

We see this in technology architecture just as much as any other discipline. When approaching a complex problem it is very useful to discover the underlying principles or laws that will govern the solution. For example, from my own experience of architecting a distributed system that wanted to exhibit the following behaviour:

• Under normal conditions each control centre would cooperate with others
• In a major crisis, each control centre could operate totally autonomously
• Life criticality would require extremely high availability under extraordinary circumstances
• No single points of failure
• and so on...

As requirements they are like laws in themselves, but after much exploring of the options and pair architecting principles began to precipitate out that would go on to govern the solution:

1. Each control centre must have a copy of the entire system's operational data
2. All operational data sets would need to be able to be range partitioned
3. Each centre would be nominated master for read/write activity against a selection of partitions and only masters could action a write against the partition and issue the update message to all other centres.
4. There could be no single database recording the partition range and the nominated master of it - this is achieved autonomously through consensus (otherwise this dataset becomes a single point of failure and therefore needs to be federated to allow autonomy, but what if one centre doesn't receive an update of this lookup data and uses the wrong master? Then it has to become a n-way distributed transaction which won't work for other reasons...)
5. Messaging would be used to distribute copies of the data to all other centres
6. Messages would describe the intent of the update but not the data
7. Messages would be idempotent 
8. Non operational data sets are not included as the requirements for them could easily be managed through traditional approaches.
9. Should it become clear that a particular aspect is becoming astoundingly complex or economically unviable, remember that a combination of people and process may well solve the condition far more elegantly.
10. etc.

By discovering the principles and documenting the reasoning behind them you don't have to keep rediscovering them as you go through the different parts of the solution, you can just apply them - like design patterns. They encapsulate a great deal of consideration and can be readily reused, even by people not privy to all that reasoning.

When an architectural discussion starts going down a rat-hole or is getting lost, look to the governing principles: Stop, those messages are not idempotent! Stop, that would break system autonomy! Stop, that is not operational data!

Vista is still very new in it’s RTM form. For most people that means plenty of application installation and system configuration. This in turn means plenty of privileged writing to the registry or file system. This in turn will mean plenty of UAC dialogues prompting for elevated privilege.

Before I go much further I will confess that I came from the world of Unix and “su” and more recently I lived the Least User Access lifestyle, as prescribed by Aaron Margosis on Windows XP SP2 x64. LUA was initially painful, but I soon got the hang of it and persisted. I therefore have perhaps a slightly distorted view when it comes to LUA usability in Windows Vista...

So, back to my complaint. I am hearing a lot of IT professionals bitterly complaining about UAC in Windows Vista and hastily moving to turn it off! I am so disappointed! UAC is a part of the overall defence in depth strategy and is a mitigation against many malware scenarios. It is a pain in the backside for those coming from a world of running as admin, but UAC recedes mostly into the background over time.

Security guru Michael Howard has noted how this perception of UAC really does not compare with its reality. For a long time, Microsoft and its operating systems have been at the sharp end of much criticism for a lack of security. This is due to some failures (clearly) and also due to the number of deployed Windows machines making it the most attractive target and also the largest affected audience. However, take a look at the following links and I think you will agree that with Windows Vista shows considerable promise as a more secure OS:

· Windows Vista - 90 Day Vulnerability Report

· Surprise, Microsoft Listed as Most Secure OS

UAC is part of this success that we have all been hankering for and I think as professionals of the IT industry we should be doing everything to applaud this effort and perhaps accept that the innocent childhood of running around as admin are over and that the price we are paying for this new security conscious era is very small. And for those who claim that as professionals they can always spot a phishing attack or a naughty attachment – good for you. I hope you can also spot the zero day flaws that don’t require you to double click an attachment and proceed to assume your domain admin context to wreak havoc. You can then enjoy explaining to your CEO why you were so special as an IT professional that you didn’t need to adopt a security conscious mindset...