blogs.conchango.com

A Year in The Life of a Platform Architect – Month 8

Grace.Mollison — Fri, 03 Jul 2009 17:05:45 GMT

Well its been another interesting month albeit a bitter sweet type one mainly because some of the colleagues I’ve got to know in the short time I’ve been here are moving onto pastures new . You know happy for them but will miss them all the same.. Still I had no time to mull over that as Resourcing appear to have called my bluff and placed me onto another really cool project that ticks quite a few boxes although there is one I’d have rather stayed unchecked:

Working with beta software – check

Make use of the platform provisioning stuff I’ve been doing – check

Breaking builds – check

Dabbling in a bit of code :-) - check

Opportunity to learn even more – check

Working with a nice bunch of people – check

Unfortunately it means I need to learn to love SharePoint . I’m getting to know stsadm operations and properties much to my distress. I always felt ignorance would be bliss on the SharePoint front but I can’t win them all .

No cloud stuff but hey I can play with that in my spare time .

So what is it they’ve done this time that I’m so happy about well I’ve been assigned to work with our Agile team on the Scrum for Team System project. When told I was going to be doing this I was pretty apprehensive as our CTO was on that project but being a CTO means no time to do the real fun stuff anymore I guess . He knows so much though and I was wondering are they sure about this one? Luckily though I am working alongside James at least till I find my feet and Stuart is keeping a fatherly eye on the project anyway . So I have two über consultants around and I’m learning as well as delivering . I do have a steep learning curve on the MSI installer stuff but I have good tutors and I’m getting my head around it. I started off after a few days familiarisation doing some platform provisioning stuff for the target supported environments but I was obviously having far too good a time with that ( bit of a comfort zone really) so I’m now focusing on the MSI work as we are at a point where I can start contributing on that side without impacting the deliverables. Obviously I’ll be noticeably slower than the current incumbents on this until I get up to speed but we needed to get beta 1 out the door so me doing stuff I could just get on with and getting to know TFS 2010 made sense and it also gave me the confidence to know I may not have bitten off more than I can chew . At least with the platform provisioning piece I’ve already contributed to the project without feeling like a spare part . Time will tell but so far I feel its started well although the MSI stuff could prove a tough one I’ll keep you posted on that one .

As for the project I spoke about in the past couple of posts that’s been parked at least for the moment. So yep it is another thumbs up and TFS 2010 offers plenty of opportunity to be ahead of the game.

Social Media Networks & Communication Services: When do you engage and How?

Derek.Dunlop — Fri, 03 Jul 2009 11:29:30 GMT

At what point do retailers embrace the social shopping sphere? From reviews, comments, payments, marketing and brand voice, where and how do they begin? There are so many social networks and sites here today gone tomorrow that retailers and brands are concerned that there either too late, too early or will waste their money on something that could potentially damage their brand and reputation! Equally they feel that they are missing out on this new ‘Social Shopping’ (Look at Dell who have attributed $2million in sales thanks to Twitter) and would like to get their brand and company in there but in the most effective and timely way.

Brands and retailers do need to be aware of the burgeoning ‘social space’. It can be as it is a vital lifeblood to support their overall positive position within the marketplace, helping them to be seen as forward-thinking and innovative. It is also the vehicle that may drive many businesses forward in the coming years. Some may look at networks and such as Twitter, Facebook or MySpace and think they are niche plays – and it does not really affect them. If brands want to meet their audience, they need to connect with them in the correct forums. This is not the case. For example 70% of Twitter users in the UK are over the age of 35 (comScore Media Metrix) Retailers should be aware of and monitor the social ‘chatter’ that resonates around their brand. They have to understand where the chatter is, how to track it and ultimately what to do with it. This responsibility typically sits with marketing or brand teams as they are often the official gatekeepers for the product or service. Retailers can then decide if there is a need or desire to connect with the network or forum – but what they do and how they interact is equally important:

1. Sit there and do nothing = can sometimes imply apathy and lack of understanding to the community

2. Push Push Push = using the community solely for gain and monetary advancement

3. Defend & Answer = defensive stance that shows again that the brand is only interacting to trouble shoot issues

A more strategic approach is to observe what’s happening on the networks and look to PULL rather than PUSH – by viewing these communities as a channel to discuss / share / collaborate and improve their business. That in turn will start to create trust and respect from users. This is the social commerce that is becoming more prevalent in many industries today. As far as monetisation is concerned, this is a new economy and so it is too early to put a firm monetary value on its aggregate worth. However, in terms of positive brand visibility, its value is easy to see.

Whether you enter the ‘social space’ now or in the future, it does not make a difference - the Twitter of today could become obsolete next year. What is key is that you are aware of the most relevant social media, and ensure that you have the most the most appropriate level of interaction with them. Look within your company to see who could be a good spokesperson(s) to help channel the discussion and not be seen as an ’invasive intruder’ that is there to spam and sell.

Top Tips for the embracing the social shopping sphere:

1. Look & Review the social sphere – who is on there, what is the nature of the communication, are competitors on there, what are they doing, what is the reaction, is my brand / company being discussed?

2. Decide & Act after “Look & Review” where your participation level sits – passive, active, influencer - where this will sit within your business, who sets the tone of voice etc..

3. Pull not Push - Remember that this is not necessarily a PUSH marketing vehicle but gathering new currency of social commerce that you will want to keep and maintain

4. Trust & Reward – If you are an active participant trust the voice of your network and reward them – that can be through various methods and means – could be tips, help, advice

5. Innovate - Look to go beyond the traditional methods and where this new found network can benefit your business – product reviews / brand placement / services etc

As Twitter is the media darling just now much more so that Facebook or MySpace, it’s interesting to note the activity on that medium. Of course Twitter in all essence is NOT a social network per say but in fact a communications network (as we have seen with the breaking news recently) with social features (friends & followers) While at the same time it’s also very interesting to see who is actually responding and building up a reciprocal network – by how many they follow

It is supposed to be uber cool if you only follow a handful of cool people – but then there is the perception that you’re not really listening and you are there as a “push” medium only.

Twitter is still in its infancy and will start to see more with Twitter over the next 18 months (but at the same time towards the end of the year – there will be a new kid on the block that will take the social commentary even further!!

The PC is dead long live the NC (in the cloud)

rory.street — Fri, 03 Jul 2009 10:29:49 GMT

Cast your mind back to 1995 when the chairman of Oracle, Larry Ellison stated “The personal computer is a ridiculous device”, later followed by “It's too hard to use, too powerful, too costly...” Larry Ellison was off course championing what he thought was the PC’s replacement the NC or Network Computer. The idea behind the NC was not a bad one, everything was stored on centrally held servers (on the Internet) where you didn’t have to worry about how it was going to be backed up and you could access your information from any NC device no matter where you were in the world. The problem was his idea was probably 14 years to early and since then the PC has become slim lined and very cheap.

Today we store a lot of our information online (or dare I call it the cloud?). As consumers we store our photos on flikr, backup up our files using BT Digital Vault and Microsoft’s SkyDrive, we download and store our music on the Internet, we watch TV and films over the Internet and we log into our PC’s or servers at home using programs such as GoToMyPC. Without realising it PC’s started to fulfil the role of Network Computers with the advent of faster broadband connections. But it is not just PC’s that get to play in this new interactive world there are a multitude of devices that access the Internet daily such as Smart Mobile phones, games consoles, set top boxes and even common house hold appliances.

As things are going it appears more and more of the things we do on our computers will be hosted online. The companies providing us with these services will need to scale to meet these demands and to help them meet these demands, is my favourite buzz word of the year “the cloud”. As a consumer I am not too worried about where my data is stored or where my services are provided from as long as it is secure and I can get to my data. One of the biggest problems for services and goods companies is the age old problem of anticipating demand. If I overspend on my infrastructure to supply demand I have costs I will find harder to recoup because demand was not as high as I anticipated. Demand will not always be consistent and so as a company I would like to have the resources on tap I can pay for when I need them instantly, without having to fork out for large infrastructure I will only use intermittently.

Eventually if (and this is a big if) cloud computing takes off in a big way we may find that we no longer need the benefits of a powerful PC. Soon the very desktops we use could be stored in a “Cloud Computing Container”. No longer would we need to worry about upgrades to our operating systems, performance or the software we use - all of this would be taken care of for us. All we would need to access our desktops is a simple Internet enabled computer or dare I say Network Computer?

Chris Anderson talks Free at the ICA

dean.wilson — Fri, 03 Jul 2009 09:02:00 GMT

Yesterday lunchtime I went along to the ICA to listen to Chris Anderson, Editor in Chief of Wired and author of The Long Tail, talk about his new book Free: The Future of a Radical Price.

The book is already causing waves. Anderson’s Conde Nast stable mate, Malcolm Gladwell has challenged many of Anderson’s key arguments in a review of the book in the New Yorker. Subsequently Seth Godin has rallied to Anderson’s defence. And no doubt the debate will continue.

But back to the ICA…

The event was a fairly relaxed affair, there was a brief intro to the book and a QA session. However, Anderson didn’t really provide any deeper insight into the arguments he outlined and the audience’s questions were fairly anodyne, but to be fair it was very hot and most people probably had the ICA bar and cold beer on their minds.

However, a few points did make it to my notepad. Here’ my interpretation:

In the ‘Freemium’ economy we need to think about the ‘Pet for our penguin’

Club Penguin, if you don’t already know, is a social networking site for kids. The basics i.e. a Penguin and an igloo are supplied for free, and that’s all you really need to participate. But if you want to see your stature within the community rise, say by buying a pet for your penguin, you are soon going to have to fork out for a subscription. Time for mum or dad to get the credit card out.

The point is that anyone dealing in digital at least will to have to get used to giving things away for free and find ways to profit, through premium stuff, from the attention and engagement the free stuff generates. Take the example of music. Like it or not, musicians are powerless to stop there music changes hands for free. The smart acts are exploiting the publicity there music generates to sell a much broader experience, through touring, merchandise, books, box-sets etc. Really smart acts, such as Radiohead are effectively giving their creative efforts away (and adding to the free stuff with video mash-ups for instance), purely in order to extend the experience.

The only distinction consumers are making is whether or not content is relevant

In a digital world were content is essentially free (something Gladwell, argues against in his review) consumers are less concerned about whether or not the content they are accessing has been produced by an expert, a journalist, or a peer than they are about the relevance of the content to their particular need or interest at that moment.

The smartest people don’t work for you

Anderson argues that the future for journalism and journalists will be about persuading and shepherding the best people, writing elsewhere, to write in a particular place. For free. To me, this seems to link with a greater desire for the filtered and curated content. When everything is free who decides what is worthy of attention? It’s seems like editors still have a place in the Anderson’s vision. The point made by Gladwell is that if you can pay for the editors why can’t you pay for the writers?

True to his belief in the Freemium economy, Anderson has ensured that free versions of the book will be available under Creative Commons, mainly by negotiating sole rights to audio versions of the book. The full version (six hours long) will be completely free. The abridge version will come at a premium – time is money after all.

So in summary – some interesting arguments but I will watch with interest the debate sparked by Gladwell’s review. Funnily enough I left the ICA reminded of a recent reading by Gladwell of his latest book Outliers, which also left me wondering if the only thing I got in return for the ‘premium’ ticket was watching the author read their own book on my behalf.

DRY SQL

jamie.thomson — Thu, 02 Jul 2009 21:05:00 GMT

I recently inherited some SQL that someone else had written and had the job of “tidying it up” before it gets pushed out to production. Here’s a slightly simplified (yes, simplified) version of that SQL:

   1: --options

   2: select    asset_class

   3: ,        case    when volume > 0 then 'profit'

   4:                 else 'loss'

   5:         end as profit_or_loss

   6: ,        customer

   7: ,        sum(volume) as total_volume

   8: from    t1

   9: where    asset_class = 'options'

  10: group    by

  11:         asset_class

  12: ,        case    when volume > 0 then 'profit'

  13:                 else 'loss'

  14:         end

  15: ,        customer

  16: union    all

  17: --swaps

  18: select    asset_class

  19: ,        case    when volume > 0

  20:                     then 'profit'

  21:                 else 'loss'

  22:         end as profit_or_loss

  23: ,        customer

  24: ,        sum(volume) as total_volume

  25: from    t2

  26: where    asset_class = 'swaps'

  27: group    by

  28:         asset_class

  29: ,        case    when volume > 0 then 'profit'

  30:                 else 'loss'

  31:         end

  32: ,        customer

The SQL basically takes data from two tables (t1 & t2), aggregates each, carries out some inline expressions (for profit_or_loss) and finally unions it all together. On the surface it looks fine but there are a few problems here, namely that there is a lot of repeated code; it violates the principle of don’t repeat yourself (DRY) which preaches “single point of maintenance” and “deduplication” of code. If you take the time to check it out you’ll see that identical aggregations are carried out on the two datasets (lines 7 & 24) as are the same conversions for profit_or_loss (lines 3-5 & 19-22). Not only that but we have expressions appearing in both the SELECT clause and GROUP BY of both halves of the query (lines 12-14 & 29-31), another violation of DRY.

A bit of refactoring is called for. First job, eliminate those expressions for profit_or_loss which appear in the GROUP BY clauses:

   1: --options

   2: select    asset_class

   3: ,        profit_or_loss

   4: ,        customer

   5: ,        sum(volume) as total_volume

   6: from    (

   7:         select    asset_class

   8:         ,        customer

   9:         ,        case    when volume > 0 then 'profit'

  10:                         else 'loss'

  11:                 end as profit_or_loss

  12:         ,        volume

  13:         from    t1

  14:         where    asset_class = 'options'

  15:         )t1

  16: group    by

  17:         asset_class

  18: ,        profit_or_loss

  19: ,        customer

  20: union    all

  21: --swaps

  22: select    asset_class

  23: ,        profit_or_loss

  24: ,        customer

  25: ,        sum(volume) as total_volume

  26: from    (

  27:         select    asset_class

  28:         ,        customer

  29:         ,        case    when volume > 0 then 'profit'

  30:                         else 'loss'

  31:                 end as profit_or_loss

  32:         ,        volume

  33:         from    t2

  34:         where    asset_class = 'swaps'

  35:         )t2

  36: group    by

  37:         asset_class

  38: ,        profit_or_loss

  39: ,        customer

Ok cool, we’ve moved the expression for profit_or_loss into a subquery (aka derived table) and hence expressions have gone from our GROUP BY clauses … but we’ve still got DRY violations. The aggregation (lines 5 & 25) and expression for profit_or_loss (lines 9-11 & 29-31) still appear in two places. More refactoring….

   1: select    asset_class

   2: ,        customer

   3: ,        sum(volume) as total_volume

   4: from    (

   5:         select    asset_class

   6:         ,        case    when volume > 0 then 'profit'

   7:                         else 'loss'

   8:                 end as profit_or_loss

   9:         ,        customer

  10:         ,        volume

  11:         from    (

  12:                 --options

  13:                 select    asset_class

  14:                 ,        customer

  15:                 ,        volume

  16:                 from    t1

  17:                 where    asset_class = 'options'

  18:                 union    all

  19:                 --swaps

  20:                 select    asset_class

  21:                 ,        customer

  22:                 ,        volume

  23:                 from    t2

  24:                 where    asset_class = 'swaps'

  25:                 )t

  26:         )q

  27: group    by

  28:         asset_class

  29: ,        customer

There, much better; the aggregation only occurs once as does the expression for profit_or_loss (which in the code that I inherited occurred four different times). Our SQL is DRY and its going to be a lot easier to maintain for whomever picks the code up from me.

You’ll notice we’ve got 2 levels of nested subqueries (aka derived tables). I make no apologies for that - derived tables are a great mechanism for eliminating repeated code and if you take but one bit of advice away from this blog post it would be this: derived tables are your friend.

Would you have refactored this the same way? Maybe you might have moved the derived tables into a dedicated view. Perhaps you might even have put the derived tables into a WITH clause like so:

   1: with    t as (

   2:         --options

   3:         select    asset_class

   4:         ,        customer

   5:         ,        volume

   6:         from    t1

   7:         where    asset_class = 'options'

   8:         union    all

   9:         --swaps

  10:         select    asset_class

  11:         ,        customer

  12:         ,        volume

  13:         from    t2

  14:         where    asset_class = 'swaps'

  15: ),

  16: q as    (

  17:         select    asset_class

  18:         ,        case    when volume > 0 then 'profit'

  19:                         else 'loss'

  20:                 end as profit_or_loss

  21:         ,        customer

  22:         ,        volume

  23:         from    t

  24: )

  25: select    asset_class

  26: ,        customer

  27: ,        sum(volume) as total_volume

  28: from    q

  29: group    by

  30:         asset_class

  31: ,        customer

Let me know in the comments!

@Jamiet

Conversion, conversion, conversion – EMC Conchango become Google Conversion Professionals!

zia.zareem-slade — Thu, 02 Jul 2009 14:26:00 GMT

This week it was announced that we, EMC Conchango, are one of the first partners to be accredited by Google in their new Conversion Professionals program. Not only are we delighted to have been selected after a rigorous process of interviews, exams and more interviews but more importantly we’re delighted that Google has got behind the whole analytics and optimisation piece.

Analytics and MVT (multivariate testing testing) are things we have been talking to clients about a lot over the last few years. And while we already provide services in this area it still amazes me that so many organisations overlook these tools – particularly in light of the economic situation we are in.

After all, if it was my website and someone told me that I could increase conversion by 2% – 12% then I’d be very, very interested! The way we talk about it is that not having analytics or MVT in place is akin to not having store managers, footfall counters or clear stock positions in your stores!

To see what Retail Week have to say about it then click here or for the full NMA article here for the full Google Conversion Professional microsite click here or to talk about the services we offer in this area then email us on Experience.Planning@emc.com

Workflow 4.0 Persistence/ Long Running Workflows

Owain.Wragg — Thu, 02 Jul 2009 09:06:00 GMT

I have recently started to look at the changes introduced with Visual Studio 2010 and .NET Framework 4 Beta 1, the first thing I have looked at is Workflow Foundation 4.0. In this post I will create a simple example which illustrates a long running workflow which persists its state to SQL Server.

The example scenario is as follows; a request requiring approval is submitted to a workflow (via a WCF Service) and the workflow waits until at some point in the future the approval for the request is received and based on whether the request is approved or not the workflow performs a difference action. After the initial request has been submitted the workflow instance is persisted and unloaded and it is only reloaded when the approval for the request is submitted.

Below is the WCF service contract exposed by the RequestService, this service is called by clients who submit both requests and request approvals.

[ServiceContract]

public interface IRequestService

{

[OperationContract]

void AddRequest(Request request);

[OperationContract]

void SubmitApproval(RequestApproval requestApproval);

}

As you can see the service contract is very simple and consists of only two operations, shown below is the service implementation.

public class RequestService : IRequestService

{

private static readonly ProcessRequestWorkflowHost processRequestHost = new ProcessRequestWorkflowHost();

public void AddRequest(Request request)

{

processRequestHost.ProcessRequest(request);

}

public void SubmitApproval(RequestApproval requestApproval)

{

processRequestHost.SubmitRequestApproval(requestApproval);

}

As you can see both operations forward the request to the ProcessRequestWorkflowHost class, this class is responsible for managing the creation, loading and persistence of the workflow instances. For this example I decided to manage the workflow instances manually in order to give me more control over the persistence of the workflow instances and the resumption of the workflow instances when approvals are submitted.

The diagram below shows the workflow that is executed to process a request submission.

When the workflow is executed the WriteLine activity is called which simply writes out the passed in request’s details (the request is passed in as an argument to the workflow) to the console application which is hosting the RequestService, then the WaitForRequestApproval activity is executed. This activity is implemented in C# as shown below.

public class WaitForRequestApproval : NativeActivity<RequestApproval>

{

public InArgument<int> RequestId { get; set; }

protected override void Execute(ActivityExecutionContext context)

{

// Create the bookmark which waits for the request approval

string name = "Approval_" + this.RequestId.Get(context).ToString();

context.CreateNamedBookmark(name, new BookmarkCallback(Callback));

}

/// <summary>

/// Callback executed when the bookmark is resumed

/// </summary>

/// <param name="context"></param>

/// <param name="bookmark"></param>

/// <param name="state"></param>

private void Callback(ActivityExecutionContext context, Bookmark bookmark, object state)

{

RequestApproval approval = (RequestApproval)state;

context.SetValue(this.Result, approval);

}

The WaitForRequestApproval activity defines a bookmark named “Approval_RequestId”, a bookmark defines a point at which the workflow waits for further information before execution can be continued when the bookmark is resumed the Callback method will be executed and the workflow will continue as normal.

In our example once the bookmark has been resumed an If activity is executed (as illustrated in workflow diagram shown above), the expanded If activity is shown below.

The If activity executes one of two WriteLine activities based on evaluating the condition (RequestApproval.IsApproved), these WriteLine activities simple write out the request Id and the approval status to the console.

Finally shown below is the ProcessRequestWorkflowHost class, this class controls the creation of the workflow instances, the persistence of the workflow instances, the loading of workflow instances and bookmark resumption.

public class ProcessRequestWorkflowHost

{

private IDictionary<int, Guid> requestToInstanceMapping;

public ProcessRequestWorkflowHost()

{

// Setup mappings between request and workflow instance id - do this to save managing

// the mappings else where

this.requestToInstanceMapping = new Dictionary<int, Guid>();

this.requestToInstanceMapping.Add(4, new Guid("56A2CA8F-3C64-485E-A0BC-A4F0E880898C"));

this.requestToInstanceMapping.Add(5, new Guid("F130B19A-1B4B-43DA-B75B-6C58F0CC6419"));

this.requestToInstanceMapping.Add(13, new Guid("3B70FAFF-5FDC-4159-836F-DFD4DFF2EE4E"));

this.requestToInstanceMapping.Add(26, new Guid("BFB619F7-2BD7-4D52-99FA-97C20FA95DCE"));

this.requestToInstanceMapping.Add(55, new Guid("2D15D97F-ABCB-45C5-AC7D-F18A64F1E2E3"));

}

/// <summary>

/// Loads a workflow instance with the specified

/// instance id

/// </summary>

/// <param name="instanceId">Workflow instance id</param>

/// <returns>Workflow instance</returns>

public WorkflowInstance LoadInstance(Guid instanceId)

{

// Setup the persistance

PersistenceProvider persistenceProvider = this.CreatePersistanceProvider(instanceId);

// Load the ProcessRequest workflow

WorkflowElement wf = WorkflowXamlServices.Load(File.OpenRead("ProcessRequest.xaml")) as WorkflowElement;

// Retrieve the workflow instance

WorkflowInstance instance = WorkflowInstance.Load(wf, persistenceProvider);

return instance;

}

/// <summary>

/// Processes the specified Request

/// </summary>

/// <param name="request">Request to process</param>

public void ProcessRequest(Request request)

{

// Get the workflow instance id to use

Guid instanceId = this.requestToInstanceMapping[request.Id];

// Setup the arguments that are passed to the workflow

IDictionary<string, object> input = new Dictionary<string, object>();

input.Add("Request", request);

// Load the ProcessRequest workflow

WorkflowElement wf = WorkflowXamlServices.Load(File.OpenRead("ProcessRequest.xaml")) as WorkflowElement;

// Create the workflow instance specifying the arguments and instance id

WorkflowInstance instance = new WorkflowInstance(wf, input, instanceId);

// When instance is idle unload and persist the isntance

instance.OnIdle = () => IdleAction.Unload;

// Setup the persistance

PersistenceProvider persistenceProvider = this.CreatePersistanceProvider(instanceId);

instance.Extensions.Add(persistenceProvider);

// Start the workflow instance

instance.Run();

}

/// <summary>

/// Submits the request approval

/// </summary>

/// <param name="approval">Request approval</param>

public void SubmitRequestApproval(RequestApproval approval)

{

// Get the workflow instance id to use

Guid instanceId = this.requestToInstanceMapping[approval.RequestId];

// Load the workflow instance

WorkflowInstance instance = this.LoadInstance(instanceId);

// Resume the workflow at the bookmark

string bookmark = "Approval_" + approval.RequestId.ToString();

instance.ResumeBookmark(bookmark, approval);

}

/// <summary>

/// Creates the persistence provider

/// </summary>

/// <param name="instanceId">Workflow instance the persistence provider is for</param>

/// <returns>Persistance provider</returns>

private PersistenceProvider CreatePersistanceProvider(Guid instanceId)

{

string persistenceConnectionString = "Data Source=localhost;Initial Catalog=SampleInstanceStore;Integrated Security=True;Asynchronous Processing=True";

// Create and open the provider factory

SqlPersistenceProviderFactory factory = new SqlPersistenceProviderFactory(persistenceConnectionString, true, false, new TimeSpan(0, 1, 0));

factory.Open();

// Create the persistance provider

PersistenceProvider persistenceProvider = factory.CreateProvider(instanceId);

return persistenceProvider;

}

The constructor creates a list of mappings between request ids and workflow instance ids; I have done this to save adding further code to manage the relationship between the requests and the workflow instances – so therefore at the moment this example will only work with a specified set of request ids (4, 5, 13, 26 and 55).

The ProcessRequest method is called by the RequestService when a new request is submitted, the method creates the arguments to pass to the ProcessRequest workflow, creates a new workflow instance adds the persistence provider to the instance (enabling the instance to be persisted) and finally executes the workflow. Note that the workflow instance is only persisted when the instance becomes idle, when the OnIdle event is raised by the instance the IdleAction is set to Unload – which specifies that the workflow instance is persisted and then unloaded.

At some point in the future the approval for the request will be submitted, the SubmitRequestApproval method is called by the RequestService passing in the request approval, firstly this method calls the LoadInstance method which retrieves the specified workflow instance from the persistence store, then the previously created bookmark is resumed therefore enabling the workflow to continue to completion.

When developing this sample application I had a few problems getting the persistence to work it turns out this is because for certain persistence scenarios you need to use the Workflow 3.5 SQL Schema and whilst for others the Workflow 4.0 SQL Schema. In short for WorkflowServiceHost usage you need to use the 4.0 schemas but for WorkflowInstance usage the 3.5 schema. Have a look at http://kennyw.com/work/workflow/350 for more information; it seems that this feature will be fixed in beta 2.

I’ve attached the solution to this post; the solution also contains a readme file which explains how to setup the persistence and how to run the application.

Note: - That this post was written using Beta 1 of the .NET Framework 4.0 so therefore changes in future releases of the .NET Framework might break the example application

WCF Error: Large amounts of data from client gives error: The remote server returned an unexpected response: (400) Bad Request

merrick.chaffer — Thu, 02 Jul 2009 08:40:00 GMT

The remote server returned an unexpected response: (400) Bad Request.

Server stack trace:

at System.ServiceModel.Channels.HttpChannelUtilities.ValidateRequestReplyResponse(HttpWebRequest request, HttpWebResponse response, HttpChannelFactory factory, WebException responseException)

at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)

at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)

at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

If you get the above error when trying to send large amounts of data from a client application to a WCF service then the following changes should be made on both the client and server application configuration files.

Client has this...

<binding name="MyServiceEndPoint" closeTimeout="00:01:00"

openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"

allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"

maxBufferSize="500000000" maxBufferPoolSize="524288" maxReceivedMessageSize="500000000"

messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"

useDefaultWebProxy="true">

<readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647"

maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />

<transport clientCredentialType="None" proxyCredentialType="None"

realm="" />

</security>

</binding>

and

<endpoint address="http://localhost/MyService.Service.Host/MyService.svc"

binding="basicHttpBinding" bindingConfiguration="MyServiceEndPoint"

contract="MyService.ServiceProxy.MyServiceProxy.MyServiceContract"

behaviorConfiguration="maxItemsInObjectGraphBehaviour"

name="MyServiceEndPoint" />

and

</behavior>

</endpointBehaviors>

</behaviors>

Server has this...

<binding name="LargeMessageBinding" maxBufferSize="2147483647"

maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647">

<readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647"

maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />

</binding>

</basicHttpBinding>

</bindings>

and

<service behaviorConfiguration="MyService.ServiceImplementation.MyService_Behavior"

name="MyService.ServiceImplementation.MyService">

<endpoint address="" binding="basicHttpBinding" bindingConfiguration="LargeMessageBinding" name="MyServiceEndPoint"

bindingNamespace="http://mynamespace.com"

contract="MyService.ServiceContracts.IMyServiceContract" />

</service>

Megastructure Datacentres

James.Saull — Wed, 01 Jul 2009 20:31:10 GMT

I suggested last year that National Geographic would probably not include Cloud Datacentres in their megastructures series. I am in Dublin just as MSFT launch their “mega datacentre”. Perhaps they should!

How cloud operating systems aren’t quite there yet

James.Saull — Wed, 01 Jul 2009 19:25:00 GMT

Shrink yourself to the size of an ant and think of a classic server as a mini data centre. Banks upon banks of memory, multiple disks, many CPU cores and even network interfaces. The OS manages all of the many processes and threads, sharing the resources amongst them and when they demand more resource it makes it available to them – on demand and apparently instantly. Importantly all the resources in my mini data centre genuinely appear as one enormous pool available to all my applications at once. My application runs in a process and doesn’t care whether it consumes one core or 8. So what is a cloud OS?

Cloud OSes do appear to be quite different because somehow I need to play a part traditionally played by the OS. When my application demands more resources I get involved. Typically I need to allocate more virtual machines or storage and quite likely I need to tell my application to extend itself into this new resource (i.e. install my application components and bring it into the pool/farm/cluster/you-name-it). Then when that demand passes I need to consider releasing those resources.

Consider for a moment too the utility model, or to use a common analogy: “cloud computing is like the power grid – on demand, just in time, scale up and down”. When I need to consume more power, the statistically multiplexed grid allows me to draw as much as I want. There are even special burst providers like pumped storage schemes to handle the gigantic surge of kettles going on after a football game. If I need a milliamp for a second or 10 amps for an hour it makes no odds to me(!).

Again the cloud seems to break this analogy. My application will consume up to the limit of pre-allocated resources and stop. The data centre might have a potential for a billion petaflops but the cloud OS can’t just grant them to my app. It is a bit like knowing that I am about to plug in my electric car and ringing the grid to provision me more current.

Once upon a time the server OS used to be quite crude but they evolved and we take their sophistication for granted now. Most people struggle to even describe what an OS does. At the moment I need to resort to special software like Appistry to help me virtualise my application so that when I auto scale up/down my Amazon machine images the application can do so too. How long before these capabilities are part of the cloud OS allowing applications to genuinely scale up and down very discretely? Will all the CPUs in the cloud data centre eventually present themselves as a single virtual resource with the cloud OS presenting abstractions, much like a regular process, as execution containers or in the cloud OS will the virtual machine always get in the way?

It feels as if the virtual machine is a convenient way of *currently* spanning large amounts of hardware, but is it a stop gap? Will a genuine cloud OS come along that behaves like a regular server OS atop a giant data centre? It will be happy to present 24TB of apparently contiguous RAM and 1000 cores to a single “process”. It will be like NUMA and move application data across the server farm to the CPU that has been scheduled to work on it. Do we need it? Are we content for the next few decades to worry about how to span virtual machines and make them collaborate and auto-scale? Is the VM the new Process?

P.S. I appreciate that in many clouds, storage capacity seems to have been the first to become a truly virtualised and discrete resource, although storage performance hasn’t (i.e. most solutions are too blunt to configure for different performance characteristics etc.)

Big Ideas vs. Small Fires

dean.wilson — Tue, 30 Jun 2009 19:23:00 GMT

Every once in a while you read something that effortlessly communicates the essence of a whole load of jumbled-up thoughts that have been looking for a form of expression, that makes you think ‘That’s it! That’s what I wanted to say!’

Such moments for me are mixed with happiness at the thought of discovering someone who is on the same wavelength, and slight frustration that I wasn’t able to blurt out such a succinct communication myself.

Anyway, 10 minutes before writing this post I had one such moment while reading the following:
http://anaandjelic.typepad.com/i_love_marketing/2009/06/the-end-of-the-big-idea.html

For me, the concept of Small Fires neatly summarises the frustration I had at the tale end of my previous career in marketing and my subsequent decision to seek the company of people who deliver brand essence through doing.

At this point I would love to link to a video counterpoint. It involves the MD of a respectable UK marketing agency trying to get across some similar thoughts*. Only problem is he didn’t quite manage to un-jumble them, and confused matters with a badly thought through war metaphor.

But it seems that the ‘conversation’ as ended with the video’s removal from brandrepublic.com

*To be fair, they probably weren’t his in the first place.

Viva Cloud! Viva!

simon.munro — Tue, 30 Jun 2009 13:48:48 GMT

Lets face it – most IT systems don’t really work as well as we would like. Surely by now we would have figured out how to build systems where you don’t have to have a different username/password for ever system accompanied by different ‘forgot password’ functionality for the same? Surely by now someone would have figured out, and packaged, a way to capture time easily, qucikly and effectively instead of that gawdawful ERP system that I am forced to use? Surely someone could have fixed Outlook by now so that you don’t poke with the mouse screaming ‘What are you frikkin’ doing now, stupid email programme!?’. And that is just for simple user oriented tasks. What about the fancy enterprise package that requires you to run reports from three different places and paste them into Excel so that it can be ‘adjusted’ and so that somebody can make sense of the data? What about core enterprise systems that are so bad at interfacing that it seems like the only way to get data in is through their painfully slow proprietary interface that feels like it is only one generation away from betamax tapes? How many systems are out there that barely function and are kept going like puppies on a leash by overworked and irritated system operators.

Depending on your point of view this could be a good state of affairs or a bad one. Vendors of software that is at the core of the enterprise are interested in protecting their investments and annual license fees, so a system that needs a swarm of consultants and a good dose of fear that it may go wrong is, at least to them, a good thing. Yet others (customers, users or technical people) may view that the same problems and the same mistakes that are being made again and again should, at some point, stop. Unfortunately the utopia of redeveloping all enterprise software (like a regular fresh Windows install) to make it all shiny and new is never going to happen, at all or in part – there is just too much there. There is mainframe era software, ERP systems based on pre-web architectures pushed in to enterprises to dodge Y2K, dotcom era VB applications and badly designed databases, websites that still support old browsers (and the browsers themselves), hundreds and thousands of Excel spreadsheets (with macros and formulas written by long departed users) which are pretty much used to run the business. All of these systems are costing a fortune to maintain and the developers, whether in-house, consulting or large vendors are rushing to add features to the existing applications and architectural reworks seem to yield little (initial) benefit. An ERP vendor may need to commit $100 million to redeveloping one particular package that may not result in more license sales and not only that money does not guarantee that the final system will be better, but the competitors may be racing ahead bolting features onto their systems while a development team is tied up in architectural rework.

Some of us are trying to change this pattern and may have been striving towards the vision of better software for years – inching closer and closer as time ticks by. If I just think about the people that I work with; we have the agilists, the *DDers, SOLIDers, service pushers, design by contracters, the UXers and others - all are trying in their own way to push the engineering, architecture, design and interaction envelope to get rid of the crap systems out there that our customers, and as users ourselves, have to deal with daily. We act as (dare I use the analogy?) as terrorist cells that plant our projects as IED’s, hoping to rock the establishment – sometimes getting media attention and starting a revolution and often leaving little more than a small crater on a rural road that eventually gets paved over with Excel spreadsheets.

One of the reasons why I latched on to the cloud was that I saw patterns in developing cloud applications that in some ways nudge and in other ways force developers of cloud oriented solutions to think things through in a way that would ultimately result in better engineered and implemented solutions. The cloud forces us to deal with concurrency, consistency, lack of physical access, decoupling, contracts, reliability, peak demand, security, message orientation, queues, data performance and a long list of things that we generally know about but often choose to ignore in our existing architectures and projects in exchange for pragmatism and a belief that ‘good enough’ is, well, good enough.

While my views on cloud computing are broader than many of the vendors out there, I have been fairly supportive of technologies which may not technically or academically be defined as cloud but, by implementing some of the newer technical memes could at a push be branded as ‘cloud’. After all since ‘the cloud’ is ill defined and without consensus, anyone can put a ‘cloud’ sticker on a product and proclaim it to be so – a few striking philosophers will not change the market forces. So I have been openly supportive of the private cloud which seems, to me at least, to be a good platform for vendors to re-badge their existing products and sell them back into the same enterprise data centres that they always have. The technology that supports the private cloud, using virtualization and so on, should ultimately find its way onto the public cloud where I am hoping that the changes will really be taking place.

However, it seems that the private cloud package pushers are wrestling ‘the cloud’ term from broader use and moulding the principles to suit their ~~world domination~~ product roadmaps. It is to be expected as the marketing money and attention, at least in enterprises, it not to spend vast amounts of money turning architectures upside down for debatable benefit, but rather spending money with trusted vendors and established systems while trying to eke out as much performance and efficiency as possible. So when I hear pro-privates saying thinks like ‘applications <need to be> re-thought, re-architected and re-written to take full advantage of the cloud <is> complete and utter nonsense’ , I completely understand where the message comes from but I slump my shoulders as I see another opportunity to build better systems slip by. The vision of the cloud, to me, is not to have a bunch of Excel spreadsheets now running in a VM that lives in the cloud – I want to do away with those spreadsheets, and possibly the system that exported them in the first place – altogether and forever. I don’t believe that the cloud is about taking badly designed databases, that have to have three full-time DBA’s tweaking, tuning and coaxing them along, moved to ‘the cloud’ so that those DBA’s can now work from home, or for less, or whatever the promised benefit may be.

I will be the first to admit that my views don’t lend themselves towards a packable software product, I gave up on that a while ago when the products that I was involved with were so awesome and game-changing that the effort to sell and explain them, by forcing a mind-shift, was so high that it was not viable. I understand how the commercial needs of selling product and the desire to make money in this quarter need to be balanced against the desire for engineering elegance and people learned long ago not to bring me along for product pre-sales. However, despite all my understanding, I personally and professionally believe that the previously mentioned time recording application that I use should not be hosted in the cloud – it should be dragged out, blindfolded and shot. Maybe it will… when the revolution comes.

Simon Munro

@simonmunro

“The ear is totally neglected. Everything is centred on the eye”

Richard.Wand — Tue, 30 Jun 2009 11:33:00 GMT

I borrowed this quote from Daniel Barenboim’s talk on the Nature and Power of Music. Most technological experiences, especially online, have been primarily visual experiences. The web gives us a visual workout but few online experiences stimulate our hearing to create fuller sensory experiences. With the Microsoft Surface interaction paradigm it’s time to prepare for an audio assault as sound is getting the consideration it deserves. Interaction Designers are digging deep in to their auditory and visual toolkit as sound gets crafted in to the Surface experience. The ear is back in vogue.

OK, so the gaming industry has been treating our senses to full immersive experiences for some time but most other technological experiences have relied heavily on visual alone.

Surface doesn't just encourage but mandates a fuller sensory experience. But are agencies ready to build auditory experiences and are clients ready to pay for sound design?

If you can communicate the return on investing in sound design to your clients then they will pay for it. Sound is not there just to titillate the senses. It is used to enhance a brand, stir emotion, create immersive experiences and communicate information. When users manipulates digital content, information is revealed through sound, visual feedback and movement. Sound is not simply about providing a fuller sensory experience but acts as another information layer.

I recently met with Andrew Diey, Founder & CEO of Radium Audio, a leading music and sound design company that have experience in designing sound solutions for Surface experiences. Does Andrew feel that clients are ready?

Andrew doesn’t see such digital auditory experiences as a huge leap. Advertisers know the importance of sound to enhance a brand. This is not about introducing something new but transferring a proven art to a new paradigm. Andrew believes clients are ready but it’s down to agencies to communicate the value of sound. Andrew suggested agencies establish a manifesto which defines the terms on which they approach immersive experience design. The manifesto should outline principles, policy and action plans for future work in immersive experiences. Most agencies have a design manifesto but does it include sound? If you want your client to buy in to sound design then you need to plan for it from the beginning. The task of getting your client to understand the value depends on your belief and recognition that sound is critical to the delivery of a full sensory experience.

Are agencies ready? Yes. It hasn’t been about not having the sound design know-how but more so about not having an environment where sound design is treated with the same importance as visual design. Most agencies that specialise is online solutions may not be experienced in sound but with experienced sound designers such as Radium Audio there is no reason for agencies not to be ready. Sound is part of an interaction designers domain to define audio feedback that complements the interaction and visual design. The brief is then given to the sound designer, internal or external, to weave their magic and create seamless experiences between the audio and interaction.

At EMC Conchango we have in-house sound design capabilities but we are not professional Sound Designers. For superior production and bespoke sound design we look to work with professional sound designers. We employ experienced Art Directors and Creative Designers to achieve our creative vision. Our approach to audio is the same. To craft an immersive auditory experience we need to work with the best.

So there is no reason why we should not be entering a new sensory era of compelling and immersive experience that combine sight and sound. It’s now up to us to be heard...

Azure worker role development tips

jamie.thomson — Mon, 29 Jun 2009 13:29:41 GMT

Just lately I’ve been doing a lot of messing around with Windows Azure the main focus of which has been exploring Azure storage (tables, queues, blobs) and how those things can be manipulated from Azure worker roles. One output from that “messing around” has been my Tweetpoll application (read more: Tweetpoll my first Windows Azure application is live).

Since then I’ve found some blog resources that would have been useful during development had I found them earlier so I’m linking to them from here mainly so that I can easily find them again but also so that other people might be made aware of them because there’s some really good content here.

Firstly, Dom Green has developed a library that provides rich application tracing capabilities on top of Azure’s bog standard logging mechanisms. Read more at AzureTrace … managing all your Azure Logging Needs. I’ve put together something slightly similar myself recently but its not as feature rich as Dominic’s and, as anyone at ~~Conchango~~ EMC Consulting will tell you, I’m no .Net expert by any stretch of the imagination so I’ll be working to put Dom’s library into my own stuff forthwith.

Secondly, Dom linked to a great series of articles from Neil Kidd which cover how we could (and should) build reliable Azure worker roles. Those links:

Thank you Dom. Thank you Neil. There’s some really useful material here that I’ll be devouring constantly over the coming weeks and months.

@JamieT

Developing the Smart Metering Proposition

dean.wilson — Fri, 26 Jun 2009 08:51:00 GMT

I’ve been really impressed by the work a small team here has been doing with Microsoft on Smart Metering, and it has really got me thinking.

I was discussing my interest in the subject over a pint with a friend last week. I consider this friend to be very technology aware (he’s generally ahead of me in terms of technology adoption at least) but to him smart metering was a novelty.

Smart metering is already happening. It’s big in the states but is starting to take hold in the UK too. You can purchase a smart meter for electricity from Eon (amongst other places) or if you fancy something a bit more ‘designed’ try looking at DIYKyoto.com. There has also been a fair bit of press on the subject.

Smart metering, along with Nike+, Wii fit and Fiat’s Ecodrive, fits in to the growing trend of Life by Numbers, or as I like to call it Stats for Everyday Folk. But it seems that there is still some way to go before it becomes a talking point in the same way that these other examples have. It’s still yet to reach its tipping point.

Working on concepts to develop Smart Metering through more sophisticated interface design and functionality is definitely a great way to raise its profile but we also need to support this activity with some other thinking.

Firstly, I think we need to focus on the essence of what is attractive about smart metering, which is fairly close to the essence of Nike+, i.e. personalised feedback. Returning to the conversation I had in the pub, for my friend the mere fact that he might be able to monitor his own energy usage in terms of what it was costing him in real-time, and instantly be able to do something about it, was enough.

Secondly, we should also be thinking about the context in which Smart Metering will exist in the next five to ten years. Energy providers have been set the target of rolling out Smart Metering to every home in the UK by 2020. By that time will the ‘Millennials’, the kids who have known a world without internet access, be paying the bills? And how? Will they have more of a relationship with the Smart Meter in their lives? Or will it still just be a box under the stairs like it is for us?

And thirdly, as we develop concepts for Smart Metering we should think more like marketers and consider the proposition we are developing, and the benefits it brings to consumers, as well as interfaces and functionality.

By doing these things we can better understand the role that providers have to play in facilitating the relationship that consumers have with the numbers being generated. And for providers who are concerned about loyalty and engagement that can only be a good thing.

Living by numbers

Experience.Planning — Thu, 25 Jun 2009 18:43:00 GMT

An interesting article from Wired on how Nike ‘Unleashed the power of personal metrics’ is highlighted in our latest Experience Planning newsletter.

It’s worth taking some time to read this article. Not only because it is a cracking good yarn told well, but because it:

Highlights how insight in to human behaviour really helps to explain why simple ideas become remarkable
Picks up on a trend that is likely to grow and have a considerable influence on the ways consumers relate to brands digitally simply through exposure to the numbers generated by day-to-day activities

Relating to insight in to human behaviour there are several powerful insights:

5 is a magic number – It only takes five runs with Nike+ and the runner is hooked on what their data tells them about themselves

People change their behaviour, often positively, when they are being observed, whether it is by a boss a community or even just ones self (self-tracking)

Personalised feedback is immensely powerful in terms of its ability to create positive feedback loops

But it is the observed trend of Living by Numbers that is potentially more interesting. Alternatively we might refer to this trend as Stats for Everyday Folk.

The idea of analysing data to understand and improve performance is nothing new (and by the way we have a whole team dedicate to the subject). What is new is the emergence of tools, software and hardware, that expose the numbers behind the things that we do on a daily basis in ways that help us to see if what we are doing really does match up to what we would like to see happening.

As well as Nike several other brands have started to plug-in to and explore this trend e.g. Wesabe in the financial sector, Nintendo with Wii Fit, and Fiat with the Fiat 500 and Ecodrive. Currently a small team within EMC Conchango is working with Microsoft to explore the development of Smart Metering technology and churning out some really innovative concepts that may well change our energy consumption behaviour in the next 10 years and beyond (check out Julian Harris’ recent blog post).

These examples are all genuinely interesting in their own right, and herald an exciting step change in the way that we relate to everyday life and the way that brands facilitate that relation. But we should probably note about these examples is that their subject matter lends itself well to generating numbers e.g. distance travelled, money spent, fuel consumed.

So having identified the trend let’s set ourselves a challenge - while we are on the look out for ways to positively expose numbers and generate personalised feedback let’s turn over a few stones and see if we can’t expose the numbers hiding in some less obvious places.

SDS is an Option for Azure Persistence

simon.munro — Thu, 25 Jun 2009 15:28:52 GMT

It was pointed out to me recently that my critical position on SQL Data Services and support for Azure in general are inconsistent, so I thought that a bit of clarification is required.

SQL Server is my database of choice and have been using it since version 4.21, which was long before it became mainstream and I even spent a couple of years on Oracle to make sure that I had broader database experience and breathed a sigh of relief when I returned to the SQL Server world. As a fan of SQL Server I am really supportive of any efforts by Microsoft to put SQL Server into the cloud and was very supportive of their SDS initiative; imagine having a database platform that was both cloud capable and on-premise – it would be the only one in the market. As information has been made available about SDS, it seems that it has hit the scale-out problem of the SQL (relational) model and has no way of managing consistency and partition tolerance easily (at all?). The cloud idea of scalability by adding additional nodes as required has been lost on the SDS team and they seem to remain quiet on the issue (my question on the forums from April 2009 is still unanswered).

Now I don’t have a problem with the (lack of) scalability of SDS per se, rather the misleading marketing and evangelising that SDS is a relational database for the cloud and the implication that you develop as you normally would for a SQL database, without clearly outlining the risks associated with the lack of simple scalability. In a Tech-Ed video earlier this year members of the SDS team jovially discuss the approaches to handle scalability on SDS which comes down to two things 1) Database sharding and 2) Microsoft is working on tools/frameworks to make it easier to shard a database. In my opinion (SQL/Relational) database sharding as a starting point is bad design (you get pain and suffering without the ACID benefits of SQL) and waiting for Microsoft to develop some tools and frameworks is hopeful and risky.

Theoretical and academic discussions aside there is one simple problem with building an Azure app that uses SDS as its primary persistence mechanism. If your business gets mentioned on Oprah and suddenly makes it big, it will hit the wall. You can spin up lots of Windows Azure Hosted Services but only one SDS service and the load on that lonely stressed-out instance of SDS will render the application useless. Part of the attraction of the cloud in general and Azure in particular is the ability to lower startup costs by hosting on the bare minimum and being able to scale as needed and on demand (almost). If you do hit the wall on SDS you have two options 1) a rewrite of the data access layer to use sharding or the mythical framework that is on its way or 2) move the entire application on premise (which may be difficult because of the dependency on the Azure fabric) – neither options will be quick and easy to implement and opportunities will be missed.

The Azure platform (hosted services and storage) does not have the same scalability problems and is (seemingly) engineered with regard to scale-out as has been learned in stateless web farms and message oriented systems over the years. Azure also forces developers to architect their applications in such a way that the scalability is built in by placing restrictions on access to disk, other processes and removing most of the urge to implement cloud unfriendly practices. The focus on worker processes, queues, Azure storage and RESTful storage styles biases implementations to more service oriented styles which, almost by definition, can handle scalability. I find that this is what is the most attractive about Azure – provided we can get developers to think a little bit differently about how to process data, Azure provides the platform for massive (and painless) scalability.

In all the cloud discussions, not only at Microsoft, persistence is the elephant in the room that everyone seems to ignore. Current applications and architectures have such a high dependency on ACID data operations and all of the goodness that comes from using a SQL database. Although it is generally known that SQL databases don’t scale out very well it seems that the competitor products are immature (at least in addressing a larger problem space) so they rather keep quiet and don’t point fingers at the entrenched database vendors who, in turn, don’t want to highlight scalability problems. My issue is that these problems are ignored at the cost of the customer who eventually comes across problems that everybody seemed to know about the whole time. So it is imperative that Microsoft talk openly and honestly with customers about the architectural considerations that need to be made on Azure and not just target small businesses who don’t have the skills in-house to ask the uncomfortable questions.

I have no doubt that Microsoft has the engineering skills to provide tools, technologies and frameworks to build cloud oriented data storage mechanisms, even if it is on a SQL model (Madison is, after all, a data sharding architecture). There are clever people like Pat Helland who has a lot of experience on building distributed systems (although he is working more with unstructured data now – I think). I sincerely hope that Microsoft gives the engineers a chance to build what is needed and not just leave it up to the marketers and the customers – after all the potential technical barrier is far greater than whether or not Outlook renders html with Word or not.

So, if you are a small business and evaluating Azure and SDS, take note of the following:

Data consistency (as offered for free by SQL databases) is not always needed.
Any Azure application should make use of a number of persistence mechanisms depending on the need

SDS – where consistency is required
Azure storage (Tables/Blobs) for high volume and high throughput operations
Cache – to reduce the load on the primary persistence services
Client side data – particularly with fatter clients like Silverlight

Azure applications should make use of worker processes as much as possible and queue updates/requests against the storage mechanism if possible
Sharding should be considered as a last resort and should not be generally applied.
Considerations should be made for cloud and on-premise storage – stale and historical data can (and should) be moved to less congested storage mechanisms.

Disclaimer: My opinions are formed by experience and publicly accessible information. I have no access to MVP, partner or private beta materials and Microsoft (for obvious reasons) doesn’t talk to me directly.

Simon Munro

@simonmunro

“The Authentication Service is Unknown” error after hard disk switch

rory.street — Thu, 25 Jun 2009 09:32:48 GMT

A few days ago my Dell Latitude D830 suddenly died on me and would not turn itself back on. The IT department removed my hard disk and placed it inside a new D830. All seemed to work perfectly until I rebooted and started to get the above error. I was unable to connect to the network, certain programs wouldn't work and I wasn't even able to look at the event log to see what was wrong!

So taking out my personal laptop I Googled the error and came up with some rather interesting results. It appears that when Vista starts up, if any of the programs hang or fail it can lead to some pretty strange behaviour. The first error I received was the "People near me service has failed. A system call that should never fail has failed". It appears the culprit was the winsock service and the fastest way to solve the problem was to open a command line window as Administrator and use the following command.

Netsh winsock reset

Reboot your machine and like magic, the problem is solved.

You can find more about the problem here http://social.technet.microsoft.com/Forums/en-US/itprovistasecurity/thread/f4ce24aa-ae9d-4927-8b0f-52f3ddc4fbd9

UPDATE FROM the Sybase code museum

David.Portas — Wed, 24 Jun 2009 21:57:59 GMT

UPDATE FROM is dead … or it should be. Hugo Kornelis has done a good job of explaining why.

Apart from a badly thought out and non-portable syntax, UPDATE FROM suffers a notorious bug/feature. I say “notorious” but it’s probably not notorious enough because it is still common to see people get tripped up by it. If I write:

UPDATE Alpha
SET z = Beta.z
FROM Alpha
JOIN Beta
ON Alpha.x = Beta.x ;

and if Beta.x is not unique then Alpha.z will get updated with some random value from one of the matching rows in Beta.z. This is documented in Books Online as follows:

“The results of an UPDATE statement are undefined if the statement includes a FROM clause that is not specified in such a way that only one value is available for each column occurrence that is updated, that is if the UPDATE statement is not deterministic.”

MERGE handles the same situation more wisely and safely: it returns a runtime error if the join is non-deterministic:

MERGE INTO Alpha
USING Beta
ON Alpha.x = Beta.x
WHEN MATCHED THEN UPDATE
SET z = Beta.z ;

Msg 8672, Level 16, State 1, Line 42
The MERGE statement attempted to UPDATE or DELETE the same row more than once. This happens when a target row matches more than one source row. A MERGE statement cannot UPDATE/DELETE the same row of the target table multiple times. Refine the ON clause to ensure a target row matches at most one source row, or use the GROUP BY clause to group the source rows.

MERGE is also standard SQL, works in other DBMSs and frequently results in more efficient execution plans than UPDATE FROM.

UPDATE FROM isn’t originally Microsoft’s fault because it’s a legacy of SQL Server’s ancient historical roots at Sybase. But it is Microsoft’s fault they haven’t deprecated it yet. You can vote for deprecating UPDATE FROM on Connect:

http://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=332437

DB2 9.7 and Database Type Safety

David.Portas — Wed, 24 Jun 2009 21:01:00 GMT

DB2 has some new features that are aimed at customers who want to transition from Oracle to DB2:

http://www-05.ibm.com/ch/events/symposium/pdf/9_M_Obrecht_IBM_Symposium_2009.pdf

IBM are emphasising the improved snapshot isolation, PL/SQL support and packages (“modules” in IBM’s terms).

One “feature” that caught my eye is support for implicit conversion. This is a surprise to me. In most areas of software engineering strong typing is considered not just desirable but pretty essential to program correctness and early identification of bugs. I don’t think that reducing or removing type safety is a good way to improve database languages.

In SQL Server, Oracle and other DBMSs we already have to live with implicit typing (type coercion) and the problems it causes. SQL Server won’t complain at compile time if I write the following two lines in the same procedure or script:

SELECT col1 FROM tbl WHERE col2 = CAST(1 AS INT);
SELECT col1 FROM tbl WHERE col2 = CAST('2' AS VARCHAR(10));

The effect is that SQL Server recklessly and silently casts strings as numbers and won’t even give a runtime error until the data itself causes the conversion to break. Getting it right therefore requires much more effort than it would do if types were checked at compile time.

Erland Sommarskog has posted a feedback item requesting that stricter compile time checking be incorporated in a future version of SQL Server. Please rate it if you agree:

http://connect.microsoft.com/SQLServer/feedback/ViewFeedback.aspx?FeedbackID=260762

Cambridge User Group Meeting : SQL Server 2008 Night – 6th July 2008

James.Rowland-Jones — Wed, 24 Jun 2009 09:31:52 GMT

My colleague Simon Sabin is holding another user group meeting in Cambridge on the 6th July. The theme is SQL 2008 and should be a lot of fun.

If you plan to attend make sure you register – it’s free to attend it’s just so the guys have an idea on numbers etc http://sqlserverfaq.com/events/183/SQL-Server-User-Group-Cambridge-SQL-Server-2008.aspx

He is holding the meeting at Redgate HQ http://www.red-gate.com/about/Map_colour.pdf.

UK UPA Careers Event - Thursday June 25th 2009

Michelle.Flynn — Wed, 24 Jun 2009 08:25:00 GMT

Tomorrow night EMC Conchango will be exhibiting at the UK User Professionals' Association careers event.

Date: Thursday 25th June 2009
Time: 6.30 for 7pm.
Location: LBi, Truman Brewery, 146 Brick Lane, London, E1 6RU
The LBi Office is just north of the Truman Brewery overhead bridge - on the right just before the Vibe Bar (which is on your left)
Nearest tubes: Aldgate East, Liverpool Street, or Old Street

The event is for all those working in or otherwise interested in User Experience. It is an informal and relaxed event, offering the opportunity for you to network, seek talented practitioners, meet those soon to graduate or find a new role...there will be many opportunities to hear from different perspectives and get to know your peers.

The evening will feature several sponsors including EMC Conchango, LBi, Fortune Cookie, Flow, Bunnyfoot and Aquent – companies and agencies who will be displaying some of their work and will be there to tell you all about their approach and experiences and to answer any questions you have.

Introducing the Reverse Job Board

There will be a job board advertising people looking for a job. Simply take an A5 piece of card, state what skills and experience you can offer and what kind of job/ company you’re looking for. You don’t have to put your real name or contact details on there, just make sure to let us know your alias either before the event or after by emailing careers@ukupa.org.uk and let us know if you’re interested in being contacted by companies and/or recruitment consultants and we shall do the rest.

Remember to make it readable, memorable and that you tell us your alias so we can put you in touch with the right people afterwards. If you don’t have time to prepare a card before-hand, we’ll have some spare cards available to use on the day.

There will be a short introduction and discussion panel at 7pm. Drinks and snacks will be provided to keep you going so you can focus on socialising!

Hope to see you there.

Enterprise Mashups

jamie.thomson — Tue, 23 Jun 2009 22:16:00 GMT

Of late one may have noticed that I have become very interested in high-brow, generally vaporous, disciplines such as RESTful data services, data interoperability and cloud computing whilst coincident with that has been the inexorable rise of the term “mashup” in the information technology lexicon.

“Mashup” means different things to different people but to me its simply the practice of combining data from multiple places with the aim of discovering or passing on knowledge that wasn’t known before. Well hey, that sounds a lot like what I do in my day job; the main difference being that I don’t generally hear the term “mashup” being bandied about the London meeting rooms that I frequent to the same extent that it does in the funky web 2.0 and swanky startup world; the term I hear (and use) is the considerably less cool “data integration”. Fundamentally though I don’t think there’s that much difference between the two so maybe enterprise data integration people like myself have something to learn from these so-called mashup players.

One of my favourite mashup tools out there is one I’ve spoken about before – Yahoo Pipes. If you haven’t had a look at this it really is worth taking a glance. Yahoo Pipes enables you to extract data from multiple web-based data sources, transform it using a series of operations like sorting, joining, unioning and filtering before finally outputting that transformed data in one of a number of different formats; its a data pipeline for web-based data (A pipeline? Oh, there’s something else I’ve talked about before – noticing a pattern here?). Here is an example of a Yahoo Pipe: Yahoo Finance Stock Quote Watch List Feed w/Chart

Extracting. Transforming. Sorting. Unioning. Filtering. Outputting. This Yahoo Pipes thing is starting to sound awfully like ETL tools such as SQL Server Integration Services (SSIS) wouldn’t you say? They even look a little bit like each other with their boxes joined up with lines between them:

I’m now reminded of what my good friend Andy Britcliffe of Sharpcloud once said to me upon reading my blog post (and viewing the embedded video) Consuming web services in SSIS 2008 a full two years ago. I distinctly remember Andy’s words on that occasion: “SSIS is the ultimate mashup tool”! I didn’t disagree!

Most mashup tools share one common characteristic in that they invariably require someone with some technical nous to set them up in advance so that they can be used by the less tech-savvy amongst us and the same applies in enterprises as well; data is distributed by the IT guys to the information workers and this distribution of data typically takes months whereas the consumers of that data want it available in hours. In both arenas I sense a shift occurring; now the consumers of the data are being empowered to find and interrogate data for themselves and in the enterprise this is happening through the adoption of tools such as Qlikview, Omniscope and (in the near future) Microsoft’s Gemini. I find this to be a fascinating development not because it means there may be less work for me to do (admittedly that would be nice) but because information workers now have the opportunity to be much more productive in their daily jobs and I expect those who invest in learning these new technologies to be the cream that rises to the top of enterprises in the near future.

Up until recently I hadn’t been all that interested in Microsoft’s Gemini project, indeed I was very sceptical of it, but as I started to formulate some of the thoughts that I’m writing about here I began to realise how important it will be when it gets released sometime (hopefully) in early 2010. I earlier described mashups as being “the practice of combining data from multiple places with the aim of discovering or passing on knowledge that wasn’t known before” and that description fits very well with Gemini. If you don’t know what Gemini is take a look at this video:

That demo glosses over the main point I’m making which is that here we see data that is originally pulled from multiple sources and combined in a familiar place (Excel) where the end user can consume it. The person speaking in the video is Donald Farmer and he has a blog entry with many other links to Gemini resources at Microsoft Project Gemini links.

At the top of this email I also talked about how I’m interested in data services, that is data available over the web that we can consume via an API and use for our own knowledge discovery and I was introduced to such a data service just yesterday when listening to Jon Udell’s “Interviews with Innovators” podcast. In the most recent episode Jon interviewed Stephen Willmott whose company 3scale Networks has taken it upon themselves to make data held by the United Nations freely available via a data service to anyone that would like to consume it. For example, if you want to know the United Kingdom population’s annual growth rate since 1991, that data is available, for free, at http://undata-api.appspot.com/data/query/Population%20annual%20growth%20rate%20(percent)/United%20Kingdom?user_key=XXXX (you need to sign-up for a free user-key and substitute it for XXXX in order for this query to work) and is returned like so:

“Wouldn’t it be cool” I thought, “if I could consume that data inside of Excel using Gemini”, perhaps in this example to combine it with birth rates over the same period to discover if there is a correlation between the two. At the time though I didn’t know if Gemini made it possible to consume data directly from data sources so I went straight to ask the man who would know, the aforementioned Donald Farmer. I contacted Donald over Twitter and here is the conversation that ensued:

Me: @donalddotfarmer Is there a list of data sources types from which #Gemini can get data? Interested in data from web APIs e.g. undata-api.org (link)
Donald: @jamiet I'll need to check out that site in particular, but we do support Atom feeds. (link)
Me: @donalddotfarmer Ahh that's good news. How about POX/RSS? Does Gemini allow us to parse it or use XQuery? (link)
Donald: @jamiet No we don't support XQuery - we just consume Atom feeds as they come - the users can then filter and sort in Gemini (link)
Me: @donalddotfarmer OK, so Atom only right now. Looking forward to getting hands dirty, think I know what 1st feature request will be :) (link)

Lots of techy abbreviations in there so let me summarise. Gemini will be able to consume data from web services that deliver it in the popular Atom XML dialect (more on Wikipedia) which is great news and no great surprise given that Microsoft announced in February 2008 that Atom would be their XML syndication format of choice going forward (see my blog post Windows Live Dev announcements for a more complete commentary). I happen to know that the United Nations data provided by Stephen Willmott is not currently delivered in Atom format but no matter, at least things are moving in the right direction and as I alluded during my last tweet to Donald I’ll be asking for support for other syndication formats in the future.

This has turned into a rather rambling blog post so I’ll call a halt here. As always though I’d be interested to know other people’s thoughts on data services, usage of that data in enterprises or anything else I’ve mentioned herein so if you have any thoughts please leave comments in the space below!

-Jamie

Geo Data Visualization – Here and There

john.brookmyre — Tue, 23 Jun 2009 21:47:01 GMT

I have come across a visualization technique called Here and There by Schulze and Webb from Alexis Kennedy (as ever thanks), other blogs and Wired Magazine (UK, June 2009). See below what it looks like:

As mentioned in Wired, it is like street view on a roller coaster – eye level and birds eye view. It is definitely innovative but it is not immediately clear what this gives above the “old world” of Google Maps and Google Earth. It is cool but what benefits can it offer - Some ideas that are suggested on S&W’s blog are:

see through the city into the distance
real-time activities like:

traffic volume overlaid on the distant city map
taxis or other services
bottle necks
all the things you can do with Google Earth / maps

All the same it is always nice to see a new way of visualising. Comments, thoughts and ideas would be appreciated!

John

Technorati Tags: Data Visualisation,Data Visualization,Geo,Wired,John Brookmyre,Information Visualisation,Information Visualization

The 3rd Fantastic Tavern - Total Experience Design

matthew.bagwell — Tue, 23 Jun 2009 13:00:00 GMT

Following the success of the 1^st and 2^nd Fantastic Taverns, we are back for the 3^rd time.

Date: Thursday 9^th July

Time: 6:30 onwards

Venue: The Roebuck, 50 Great Dover Street SE1 4YG

It’s no longer the case that digital experience is an adjunct to marketing, sales or brand experience planning and strategies. Digital agencies have moved from the cheap seats at the back of the room to the top table, arguably providing the 'glue' that holds all these threads together.

But what follows?

The Fantastic Tavern returns to explore the emergence of Total Experience Design. By combining specialist skills in new ways, can the architects of digital experience now go wider and deeper than any other 'traditional' agency?

What does this mean for creatives as their big ideas extend into a proliferation of touchpoints?

How do clients buy the service and who owns total experience within the enterprise?

How can total experience be measured from channel to channel?

Once again, the creative meister (@mattbagwell) brings together leading exponents of this exciting discipline to fuel the debate.

Can ted really be done by one agency?

What are the practices they need to follow?

How can we help clients buy and prepare for the change in mind set required for success?

What are the skills and roles we will need in our creative teams in the near future?

We may not find all the answers, but one thing is true. Change happened and total experience design may be the best way to harness it.

If you would like to attend please email michelle.flynn@emc.com

Also we are keen to continue growing this community so please let anyone you think would like to join us for debate (as well as beers ad nibbles) know about this event.