blogs.conchango.com

welcome to the conchango blogging site
Welcome to blogs.conchango.com Sign in | Join | Help
in Search

Steve Wright's Blog

  • Using NDepend to help guide Refactoring

    In my other blogs entries I mention that I have been looking into building a Team Foundation Server Data Warehouse Adapter.  After I got my initial proof of concept version working, I started to extend it to get a list of available builds from TFS and then import the output of Source Monitor – a popular free code metrics tool.

    Now I must state that I am not a natural .NET coder.  My normal day-to-day work is based around the SQL Server technology stack. I do have some .NET coding skills but generally just enough for what is need in and around SQL Server.  So I asked my colleague, Howard van Rooijen who is one of our top .NET coders, to have a look at my code and give me some hints.

    This provide me with a great opportunity to learn from Howard about some more advance Design Patterns,  Visual Studio tips and tricks, and how to effectively use some tools I’ve not used before to help me write better code. The first thing Howard did was install the following on my machine:

    • ReSharper – to help me refactor the code, fortunately we have just rolled out ReSharper licenses to all .NET Developers!
    • StyleCop – to help remind me of the Microsoft C# Coding Conventions
    • TDD.NET  to ease the pain of running my unit tests
    • NDepend – a code metrics tool

    Howard wanted to integrate NDepend into my TFS Data Warehouse Adapter and suggested that the best way to understand the tool would be to use it to analyse my TFS Adapter code and see if it would be able to give the direction on where to focus our refactoring efforts.   

    NDepend  comes with 82 code metrics out of the box, but it’s far more than a standard code metrics tool. It comes with a bundle of best practice design rules which are written using Code Query Language – an idea so simple and elegant that I bet the FxCop team are kicking themselves for not thinking of it first. CQL allows you to write statements like:

    // <Name>Methods poorly commented</Name>
    WARN IF Count > 0 IN SELECT METHODS WHERE PercentageComment < 20 AND NbLinesOfCode > 15  ORDER BY PercentageComment ASC

    You can even embed CQL directly into your code via attributes. NDepend also has features like being able to compare different builds (Patrick Smacchia, NDepend’s developer demonstrated this feature on the recent .NET Framework 3.5 SP1 release), manage complexity, dependencies and monitor the health of your build process.

    We started by running my code through NDepend and the initial results were that my code has high instability and not at all abstracted.  I couldn’t really affect the instability as my code was just a thin wrapper around the TFS API.  It also highlight that some of my old best practices that needed updating as many of them are now redundant with C# 2.0 features (generics, static classes etc). It also highlighted performance gains I could make.

    Howard said that he was having some difficulties understanding what the intention of my code was, as he was unfamiliar with building Data Warehouse Adapters. He asked me if I could write some unit tests that would call the code and verify the output – so he could use the “Test with... Debugger” feature of TDD.NET to see what was going on with the code.

    Once I had written some tests, Howard wrote two more: one that passed in a null value and another that passed in an empty string; these, he explained were two valid edge cases for handling bad data and to my surprise both tests failed! Once I fixed both failing tests, he asked me to run the whole test suite again and this time I was really surprised as all of my previously passing tests now failed! The issue was with two helper methods that convert string based representations of number within the Data Warehouse into valid number primitives.

    NDepend pointed out lots of issues with these methods and Howard said that a simple Regular Expression could replace a lot of the parsing logic I had written, so I set about refactoring the code.

    Before:

    public static string CleanInt(string number)
    {

        string result = string.Empty; 
        bool foundNonDigit = false
        int index = 0;

        if (number == string.Empty) 
       
            return "0";
        }

        while (!foundNonDigit) 
       
            if (char.IsDigit(number[index])) 
           
                result = string.Format("{0}{1}", result, number[index]); 
                index++; 
           
            else 
           
                foundNonDigit = result.Length > 0; 
           
       
        return result;
    }

    public static string CleanFloat(string number)

        string result = string.Empty; 
        bool foundNonDigit = false
        bool foundDecmialPoint = false
        int index = 0;

        if (number == string.Empty) 
       
            return "0"
       
        while (!foundNonDigit) 
       
            if (char.IsDigit(number[index]) || (number[index] == '.' && !foundDecmialPoint)) 
           
                result = string.Format("{0}{1}", result, number[index]); 
                foundDecmialPoint = number[index] == '.'
                index++; 
           
            else 
           
                foundNonDigit = result.Length > 0; 
           
       
        return result;
    }

    After:

    public static string CleanInt(string number)

        int resultFromTryParse; 
        // check for empty string after cleaning string 
        if (number.Trim() == string.Empty) 
       
            return "0"
       
        // if try parse works return the number in string format 
        if (int.TryParse(number, out resultFromTryParse)) 
       
            return resultFromTryParse.ToString(); 
        }

        // else return results of clean GetCleanNumberString 
        return GetCleanNumberString(number, "Integers");
    }

    public static string CleanFloat(string number)

        float resultFromTryParse; 
        // clean the number and check to see if it's empty 
        number = number.Trim(); 
        if (number == string.Empty) 
       
            return "0"
       
        // try parse the number if works the return the number 
        if (float.TryParse(number, out resultFromTryParse)) 
       
            return number; 
       
        // returns result of Get clean number 
        return GetCleanNumberString(number, "Floats");
    }

    private static string GetCleanNumberString(string number, string numberType)

        // Create a regualr patten with name groups do a macth on the number entered 
        Regex regex = new Regex(@"(?<Floats>(?<Integers>\d+)+\.\d+)|(?<Integers>\d+)", RegexOptions.Singleline); 
        Match match = regex.Match(number); 
        string results = string.Empty;  

        // if match groups are found the return the string 
        if (match.Groups != null
       
            results = match.Groups[numberType].Value; 
       
        return results;
    }

    Now all my tests passed and NDepend was happy too.

    Next NDepend highlighted some of my methods that had too many lines of code. Howard said he was having a bit of difficulty understanding what the code was doing and explained to me the notion of Programming by Intention.

    Next he paired with me and went through the method reformatting the code. He said that by reformatting, reorganising and removing redundant variables you start to see patterns forming within the code, that clumps of logic naturally come together and how you can then use ReSharper’s Extract Method refactoring to extract that clump of code into a separate method. Now instead of having a mass of hard to understand code – the method simply listed a series of steps that occurred – the intention of the method had been made clear. Howard was happy because he now understood what the method was doing and NDepend was happy as the method was now shorter and a great deal less complex.

    In my original proof of concept I had used partial classes as a mechanism to extend the main TFS Data Warehouse Adapter and implemented another partial class in order to integrate Source Monitor.  Howard pointed out that this approach wouldn’t work if we wanted to integrate a second tool.

    He said that now we had cleaned up the code and broken down the few large methods into a series of smaller methods we had surfaced the behaviour required to integrate an external code metrics tool into the Data Warehouse Adapter and that we could use ReSharper’s Extract Interface refactoring to move the definitions of this behaviour into an interface called ICodeMetricAdapter and then use this interface to implement any other 3rd party code metrics tool.

    Howard said that we had now essentially implemented the adapter pattern. Next we refactored the existing code to use the interface instead of the concrete type and we extracted the Source Monitor code into a new file called SourceMonitorAdapter which implemented the ICodeMetricAdapter interface. We added a List<ICodeMetricAdapter> to the parent object and then refactored the code to loop through this list and then call the current ICodeMetricAdapter.

    We managed to get this all done in one afternoon, just as I was packing up Howard asked me “Now, what do you need to do to add an NDepend Adapter?” I created a new class called NDependAdapter and made it implement ICodeMetricAdapter. Howard showed me that I could use the Implement Members quick fix by selecting the interface and hitting ALT-ENTER. Then all I had to do was add a new instance of the NDependAdapter to the List<ICodeMetricAdapter> and the new NDependAdapter is automatically called!

    Below is the before and after class diagrams of the solution:

    Before:

     

    before

    After:

    After

     

  • How I built a Team Foundation Server custom data warehouse adapter

    As mentioned in a previous blog I would like to explain some of the steps I took to get my custom data warehouse adapter to work.  I am going to start from the beginning just so that I have an easy starting point.  The code I show here was based upon my POC and written in C#.

    1. Create a new C# Class Library project in Visual Studio. Then add references at least to the following assemblies: Microsoft.TeamFoundation.dll, Microsoft.TeamFoundation.Client.dll, Microsoft.TeamFoundation.Common.dll, Microsoft.TeamFoundation.Warehouse.dll and System.Web. As I need to connect to the build server I added Microsoft.TeamFoundation.Build.Common.dll and Microsoft.TeamFoundation.Build.Client.dll
    2. Then specify the use of the IWarehouseAdapter interface.
    3. Implement IWarehouseAdapter.RequestStop which only sets a stop flag to true which can then be tested by the other methods periodically.
    4. Implement the IWarehouseAdapter.Initialize. This method this where I need to store objects that will need to communicate to the all systems that you adapter needs to talk to i.e.:
    string url;
    TeamFoundationServer tfs;
    _DataStore = ds;
    if (_DataStore == null)
    {
       throw new Exception("Null data store.");
    }
    url = Microsoft.TeamFoundation.Server.TeamFoundationApplication.TfsNameUrl;
    tfs = TeamFoundationServerFactory.GetServer(url);
    if (tfs == null)
    {
       throw new Exception("TF Server instance not obtained for TFS url: " + url);
    }
    _BuildServer = (IBuildServer)tfs.GetService(typeof(IBuildServer));
    if (_BuildServer == null)
    {
       throw new Exception("Build Server instance not obtained for TFS url: " + url);
    }
    _CommonStructureService = (ICommonStructureService)tfs.GetService(typeof(ICommonStructureService));
    if (_CommonStructureService == null)
    {
       throw new Exception("Common Structure Service instance not obtained for TFS url: " + url);
    }
    5. Implement the IWarehouseAdapter.MakeSchemaChanges. This method is called to see if there are schema changes that would need to be made to the data warehouse.  This is recommended to be where you actually register your changes you need to make to the warehouse i.e.: 
    SchemaChangesResult result = SchemaChangesResult.NoChanges;

    //Get the current data warehouse configuration 

    WarehouseConfig warehouseConfig = _DataStore.GetWarehouseConfig();

    //As the IWarehouseAdapter.MakeSchemaChanges can be called many times in one sessions
    //I needed to see if your fact was already created

    Fact MyNewFact = warehouseConfig.GetFact("MyNewFact");
    if (MyNewFact == null)
    {

    //Create My New Fact

    MyNewFact = new Fact();
      MyNewFact.Name = "MyNewFact";

    //Adding a link to the Team Project – I had to do this otherwise my save would not work

    MyNewFact.DimensionUses.Add(CreateDimensionUse("Team Project", "Team Project"));

    //Adding a link to the Build dimension also when use the dimension I needed to
    //find out the dimension key attribute was for when saving my data.

    MyNewFact.DimensionUses.Add(CreateDimensionUse("Build", "Build"));
    MyNewFact.Fields.Add(CreateField("measure1","int",0,"Sum"));
    MyNewFact.Fields.Add(CreateField("measure2", "int",  0, "Sum"));

    //Setting which perspective it would show under if enterprise edition of AS was in use.

     MyNewFact.PerspectiveName = "Code Churn";
     MyNewFact.IncludeCountMeasure = true;
    if (!_StopRequest)
    {

    //Starting a Transaction to rollback my changes if they fail

     _DataStore.BeginTransaction();
    try
    {

    //Adding to the data warehouse configuration xml file and saving the changes

    warehouseConfig.Facts.Add(MyNewFact);
     _DataStore.Add(warehouseConfig);
     _DataStore.CommitTransaction();
     result = SchemaChangesResult.ChangesComplete;

    }
    catch
    {
    _DataStore.RollbackTransaction();
    throw;
    }
    }
    else
    {
    result = SchemaChangesResult.StopRequested;
    }

    }
    return result;

    //The functions to create a new field and dimension link

    private Field CreateField(string FieldName, string FieldType, short FieldLength, string FieldAggregationFunction)
    {

    Field newField = new Field();
    newField.Name = FieldName;
    newField.Type = FieldType;
    newField.Length = FieldLength;
    newField.AggregationFunction = FieldAggregationFunction;
    return newField;

    }

    private DimensionUse CreateDimensionUse(string DimensionName, string UseName)
    {

    DimensionUse dimensionUse = new DimensionUse();
    dimensionUse.DimensionName = DimensionName;
    dimensionUse.UseName = UseName;
    return dimensionUse;

    }

    6. IWarehouseAdapter.MakeDataChanges which is where all the transferring and transforming of the data takes place.

    IEnumerator projectEnum = _CommonStructureService.ListAllProjects().GetEnumerator();
    DataChangesResult result = DataChangesResult.NoChanges;
    IEnumerator buildEnum;
    IBuildDetail[] buildDetails;
    ProjectInfo currentTeamProject;
    DateTime buildStartSearch = LastMyNewFactPrcoessedDateTime;
    DateTime buildEndSearch = DateTime.Now; while (projectEnum.MoveNext())
    {

    //checking to see if a stop request has happend if so then return the fact that a stop happen.

    if (_StopRequest
    {

    return DataChangesResult.StopRequested;
    }

    //Just checking the state of the project as I am not interested in deleted projects

    currentTeamProject = projectEnum.Current as ProjectInfo;
    if (currentTeamProject.Status != ProjectState.Deleting)
    {

    //Get a list of builds and the filter the out builds that I have already processed.

    buildDetails = _BuildServer.QueryBuilds(currentTeamProject.Name);
    var filterBuildList = from bd in buildDetails
    where bd.StartTime >= buildStartSearch
    && bd.FinishTime < buildEndSearch
    select bd;

    //From my new list of builds save my fact details

    buildEnum = filterBuildList.GetEnumerator();

    while (buildEnum.MoveNext())
    {

    if (_StopRequest)
    {
    return DataChangesResult.StopRequested;

    }
    SaveMyNewFactEntry (buildEnum.Current as IBuildDetail, currentTeamProject);

    //Update the last build started and date I have just covered.

    LastMyNewFactPrcoessedDateTime = buildEndSearch;
    result = DataChangesResult.ChangesComplete;

    }
    }

    }

    private void SaveMyNewFactEntry (IBuildDetail buildDetail, ProjectInfo currentProject)
    {

    Random rand = new Random();

    //Check to see if this build has been already added to fact table.

    string buildURI = LinkingUtilities.DecodeUri(buildDetail.Uri.AbsoluteUri).ToolSpecificId;
    if (_DataStore.GetFactEntry("Build Details", buildURI) != null)
    {


    FactEntry newFactEntry = _DataStore.CreateFactEntry("MyNewFact ");

    // Had to add Tracking ID by not doing so cause an error.  The Tracking ID is used to find facts
    //So need make sure that method which repeatable for the same fact entry each time.

    newFactEntry.TrackingId = Guid.NewGuid().ToString("D", CultureInfo.InvariantCulture);
    newFactEntry ["Team Project"] = LinkingUtilities.DecodeUri(currentProject.Uri).ToolSpecificId;
    newFactEntry ["Build"] = buildURI;
    newFactEntry ["measure1"] = rand.Next(0,100);
    newFactEntry ["measure2"] = rand.Next(0, 100);

    //Again create a transaction save the entry if successful commit the changes

    _DataStore.BeginTransaction();
    try
    {

    _DataStore.SaveFactEntry(newFact, true);
    _DataStore.CommitTransaction();
    }
    catch
    {

    _DataStore.RollbackTransaction();
    throw;
    }
    }
    }

    7. I need to store the last date processed so I created a property which stores the data in the datastore property bag.

    // Last changeset that was populated.

    private DateTime LastMyNewFactPrcoessedDateTime
    {

    get
    {
    String LastMyNewFactPrcoessedDateTimeStr = _DataStore.GetProperty("Last MyNewFact Prcoessed DateTime");
    DateTime lastMyNewFactPrcoessedDateTime = DateTime.MinValue;
    if (!String.IsNullOrEmpty(LastMyNewFactPrcoessedDateTimeStr))
    {
    lastMyNewFactPrcoessedDateTime = DateTime.Parse(LastMyNewFactPrcoessedDateTimeStr);
    }
    return lastMyNewFactPrcoessedDateTime;
    }

    set
    {

    _DataStore.BeginTransaction();
    try
    {
    _DataStore.SetProperty("Last MyNewFact Prcoessed DateTime", value.ToString());
    _DataStore.CommitTransaction();
    }
    catch
    {
    _DataStore.RollbackTransaction();
    throw;
    }
    }
    }


    8. Build the adapter as a DLL.

    9. Copy the build DLL into the warehouse plugins folder on the application tier.  In most cases will be C:\Program Files\Microsoft Visual Studio 2008 Team Foundation Server\Web Services\Warehouse\bin\Plugins.

    10. Reset IIS.

    11. On the application tier, navigate to http://localhost:8080/Warehouse/v1.0/warehousecontroller.asmx?op=Run and click Invoke.

    12. On the application tier, navigate to http://localhost:8080/Warehouse/v1.0/warehousecontroller.asmx?op=GetWarehouseStatus and click Invoke. Continue doing this until the status is returned as Idle. Check the application event log, to see if an exception has been thrown.

    13. Reset IIS.

    If you wish to find out what your current warehouse configuration is then run the following query against your TFS warehouse:

    SELECT CAST(wc.Setting AS XML) AS setting FROM dbo.[_WarehouseConfig] AS wc WHERE wc.ID = 'ConfigXML'

  • Building a Team Foundation Server custom data warehouse adapter

    I have been investigating a way of adding more metrics about our builds into the Team Foundation Server 2008 warehouse.  This would then allow us to measure quality of our code over time by other metrics than the just method offered by the Team Foundation Server 2008.

    So my goal was to create a new fact table and link this to build dimension.  The reason for building a fact table was the metrics as I would want to do some aggregation against them; this would mean I need to treat them as measures and measures only live in a fact table.  I also wanted a separate fact table so my adapter doesn’t impact the standard TFS adapters. 
    As my facts would be generated by builds it would mean that facts would link to the build which why I used the build dimension.  Also I found out later I had to also include the Team Project dimension as well.

    I managed to find some useful links that helped me write my adapter:

    http://msdn.microsoft.com/en-us/library/bb130342.aspx - This section covers several topics: how to implement an adapter; how the data warehouse is created and gives example on how to create adapter.  The example is only how to extend existing dimensions not how to create new objects.

    http://tomfury.spaces.live.com/blog/ - This guy was trying to do the same thing as me and created a series of blog entries explaining what he was doing and his issues he was facing and gave some code examples.  Also he has posted the following questions on the forums which were helpful:

    http://forums.microsoft.com/MSDN/showpost.aspx?postid=2464644&siteid=1
    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1644099&SiteID=1
    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2518423&SiteID=1

    There some has been some changes, as I understand, between 2005 and 2008. As I no longer code against TFS 2005 I can’t make any real comment about these.  However I found these links helpful for coding against TFS 2008:

    http://ozgrant.com/2006/06/18/how-to-list-tfs-team-projects-using-the-api/ - this to gain information about the projects that are in TFS server.

    http://notsosmartbuilder.blogspot.com/2007/09/team-build-2008-api.html - this helped me to query TFS server about the builds it understands.  As I can see there has been a lot of changes between the 2005 and 2008.  In the 2008 API some namespaces have been made obsolete (Microsoft.TeamFoundation.Build.Proxy; also a few classes in Microsoft.TeamFoundation.Build.Common).

    To see how I wrote my custom TFS data warehouse adapter please click here.

  • Could HTTP Soap / T-SQL endpoint be replaced by Reporting Services XML Reports?

    I have been working on a POC using SQL Server 2008 and Google Maps.  The POC was to use Geo Spatial technology to illustrate our Geo Spatial data easily.   I will talk about this in more detail in another blog post as I would like talk about a problem I faced while creating this POC. 

    The problem I faced was how could I get my Geo Spatial data, which was stored within SQL Server 2008 using the new geography data type, into  "static" HTML?

    So I thought that SQL Server HTTP Endpoints for SQL would be great thing to use to solve this problem.   As all I was after was xml document to stream over HTTP.   There was no business logic needed so all the .Net would be is just another layer acting as go between.  Also I didn’t what to waste time on what code which wouldn’t have added any value.

    So I created my stored procedure to produce an XML document that I found usable by using the For XML Path.  Then I started to look at how to create a HTTP endpoint for SOAP. After working through this I was presented with this message from my SQL Server 2008: "Creating and altering SOAP endpoints will be removed in a future version of SQL Server. Avoid using this feature in new development work, and plan to modify applications that currently use it."

    It was nice to see that Microsoft are giving people plenty or warning about breaking changes for next version of SQL server.  It’s a surprise that a feature is being dropped so quick, second only to Notification Services since I have known the product, but I can understand the decision as a lot of questions on how, why  &  should it be used were ask by some of our SQL Server developers.   So I took notice of Microsoft warning and tried to see if there was another method I could get my data into an XML document and streamed over HTTP without using .Net code and web services.

    All I was doing with this data was just reading for display via another component.  The data was in a structured format.   Then I thought about Reporting Services, as you can write a report and get the reporting server to render to XML.   So I quickly wrote a report to get the outputted xml into the format that I wanted.  Then in my static html I changed my http request to point to my report server with added option to render to XML.

    This approach did work and was great, as I managed to produce a page that could request data XML data and render data without any need to write .Net code.  The static html page would always remain up to date.  Another bonus was I had the report ready to show the data in a table from without having to re-write or reduplicating SQL code.  I was also then able to extend this to accept parameters so I could order the data base upon some user data from the HTML input control.

    Now this approach is not always going to work for every situation.  One reason is that reporting services security will not allow anonymous access.  So when using a windows security the report server would need to be on the same server.  A method to overcome this would be to create some forms authentication method on the report server and get your site to create the form cookie automatically. 

    Another potential problem is way that Reporting Services sends the XML data other HTTP.  The XML data is actually sent as an attachment file on the HTTP response.  Now some applications, like the side bar gadgets, might not be allowed to, or can’t, handle http attachments.

  • How E-mail the Sprint Burndown Chart to yourself or your team.

    I have been working on Scrum for Team System focusing on the reporting functionality of this tool.  One of the things I looked at was to get the reports delivered to me by e-mail.  I went looking to see if there was an interface via the Team Explorer interface, to which I found quickly there wasn’t.  However because TFS uses Reporting Services I knew that I could use Reporting Services subscriptions functionality. 

    Firstly, if not already configured, the Reporting Services server will need to will need to be set up to point to an SMTP server.  This can be done by using the Reporting Services configuration tool and updating the e-mail settings within this tool.  

    You can find this tool on your TFS server under the following: Start Menu > All Programs > Microsoft SQL Server 2005 > Configuration Tools

    To create the subscription you need to use the Report Manager  (to gain access to this you can right mouse click on the report folder within the team explorer):

    When the report manager is loaded then the root folder will contains folders based upon the name of Team  Projects. Click on the folder that contains the required reports. After the list of reports have loaded click on the Sprint Burndown report to view the report.  After the report has rendered select the subscriptions tab, once the page has loaded and then click on new Subscriptions.  Then filling the necessary details on how the report should be delivered and what report parameters the report should used when it runs.  Finally I would recommend that for the parameters select the option to pick the defaults of the reports.

  • How to tell when the Team Foundation Server data warehouse was last updated.

    While I have been working on the Scrum for Team System reporting I came across a re-occurring issue.  The issue was not knowing when our data warehouse last updated or when the next update was due to happen. 

    After some investigation, including some information from a link on one of my colleague’s blog posts, I found the answer to the issue.  In the TFS  data warehouse there is a table called: _WarehouseConfig.  This contains some useful information like when the last data warehouse was last process time, process interval and end time of each step of the process ended.  

    So with this information I updated the version report for Scrum for Team System to include the following information: process interval time, next time process is due, last time the cube was processed, last time the data in data warehouse was updated.

    I have attached a copy of the new report as it not exclusive to Scrum for Team System.  Also this report was written for TFS 2008.

  • How To: Install SQL 2005 RMO merge applications on Win64 (x64).

    For my current project I have inherited a couple of customized RMO applications. They help to create the merge publication, publication snapshots, subscribers, subscribers snapshot and synchronization sessions.  When it became time to use the application on a Windows 2003 64 bit server I came across a couple of issues when trying to synchronize. 

     

    The first problem I saw was that I was getting a class not registered error.  This was because the sqlmergx.dll was not installed by the Microsoft SQL Server 2005 Management Objects Collection install or SQL Server SDK.  The only way to get the sqlmergx.dll installed on the Server was to install a SQL Server (any edition).  This was relatively easy to resolve as I simply installed SQL Server Express. 

     

    The next error I then had was a System.BadImageFormatException message.  This was caused by the application being complied for any CPU but I was using the SQL Server Express on Windows 64.  My RMO application was therefore running as 64bit but was now referencing the 32bit sqlmergex.dll.  I therefore had to set the compile option to force it 32bit.  When a win64 version of sql server is installed the previous setting worked fine.

  • SQL Server 2005 Merge Replication Subscriber unexpectedly deleting during data Synchronisation.

    Within my current project I have been using SQL Server 2005 merge replication.  While I have been testing some issues have been raised by our testers and I would like to write about another one of these in particular.

    Data, which was entered at the subscriber, was being removed while synchronizing with the publisher.  This can happen when the data entered isn’t part of the partition for a given subscriber.  However, on this occasion we expected the data to remain in place.  After some investigation by me and my colleague, James Rowland-Jones, we found the issue.

    We noted that the affected tables were organised in parent child relationships at a table / schema level but they were joined child parent in the replication topology.  They were joined via a join filter with a partition filter on the child record.  So Customers 1 -> N Orders but partitioned On Orders with a join filter back to Customers.  The topology was created in this way to help partition the data sent to the individual subscribers i.e. a subscriber would only see customers that it had created orders for.

    To troubleshoot the problem we used SQL Server profiler to track all the SQL statements used in a synchronization session.  We found that the issue was to due to the synchronization process and how it processed the records sent from the subscriber. 

    When the subscriber has data entered it doesn’t evaluate or set its current partition mapping. The mappings are worked out dynamically during the synchronization process.  Only the data in MsMerge_Contents is sent to the publisher for evaluation.  Whilst working this out the process does not take into account the fact that a child record at a data level was a parent with the partitioning key in the topology.  During the replication process the articles were processed in foreign key relationship order so the parent data was being added first. A check was then being made to see what partition this data belonged to.  As the child data had not yet been evaluated the partitioning data wasn’t available so the topology sent a delete back to the subscriber.

    The way to resolve this was to enforce the processing order manually. We had to make a change to the database and replication.  The change to the database was to set NOT FOR REPLICATION on all foreign keys.  We also changed the topology to enforce processing order within replication. In TSQL you’d do this by setting the @processing_order parameter in sp_addmergearticle. Unfortunately in RMO this could not be set via a property or method.  I had to create a T-SQL statement to call sp_changemergearticle to update the processing order after the articles were created.

    However, there is a gotcha to watch out for with this approach as well.  As soon as one starts setting the processing order manually you need to set it for all objects.  This is because any object without an explicit number will be evaluated before those that have. So the rule is: when enforcing a processing order you will need to enforce it for all objects: Tables, Views, Procedures and Functions. 

    There are two basic approaches to setting up the processing order.  The first is to give all objects a unique number.  The second is to give all object types a unique number.  We went for the latter as we thought it’d be simpler to manage overall. It would also allow us to set big bands or ranges to allow for future changes.  However, we had to also make sure we processed the replication joins in a set order.  Therefore within the tables we gave another higher number in the range to each level of join.   Tables at the top of the replication tree were 10000 the tables they had join filters to were 20000 and so on. When we hit dependencies within objects we simply gave them another range.  We hit this with functions that referenced other functions.

    Once we’d set the processing order correctly and set NOT FOR REPLICATION on all the foreign keys everything worked and we were able to return the issue as fixed back to the testers -  which was nice. I should also point out that we did try just changing the NOT FOR REPLICATION option only but this didn’t work for us. We had to make both changes for this to work.

  • SQL Server 2005 Merge Replication not replicationing changes to the subscribers

    I have been using SQL Server 2005 merge replication within my current project.  It is a centralized Merge push topology using a custom RMO application with over 350 subscribers but that is the subject for another post…


    While we have been testing there have been some issues raised by our testers and one of these in particular is what I’d like to write about today.


    We were seeing data being inserted by the application into the publisher database but this data was not being replicated down the to the targeted subscriber (parameterized filters were being used).  Bizarrely, all the changes made at the target subscriber were replicated up to the subscriber. 


    The data which was not being replicated turned out to be new child data organized by a join filter within the replication publication.  The application is a .net smart client which is used at both publisher and subscriber database and uses a SQL Server login account to access the database. After some investigation by me and my colleague, James Rowland-Jones, we found the issue.


    Using SQL Server profiler to track all the SQL statements we traced the problem to the insert merge trigger.  The trigger was not assigning the child data to the correct partition i.e. no row being inserted into MSMERGE_CURRENT_PARTITION_MAPPINGS.  The problem was with the way the trigger checks for associated child records.  The trigger uses a generated replication view MSMERGE_REPL_VIEW_<PUBGUID>_<ARTGUID>, which rather than just performing a select on the base table to see if the data qualifies for a partition, it additionally performs a security check using an IsPalUser function. Our application user was not in the PAL and so consequently the view would return 0 results rather than the expected data.  This was certainly not what we were expecting!  The PAL is for controlling access to the publication not influencing what data is ultimately replicated.


    So to get the join filter to work as expected we had to add the application database user to the PAL.


    More on merge replication to follow….

  • MSBuild using SQL Script parameters

    I have been working with MSBuild and the Microsoft.Sdc.Tasks to deploy my current project’s databases.  Some of the SQL scripts that are executed within MSBuild need parameters to be set to create the database objects, which the Microsoft.Sdc.Tasks.Sql.Execute method supports.  To use parameters within MSbuild and SQL.Execute; firstly create an item group (<ItemGroup>) then create element of any name as this will become the name of the item that can be referred to in MSbuild.  Within the new element an attribute has to set called “Include”, then create two other elements called “name” and “value”.  Here is an example: <ItemGroup>

                <schemaName Include=”true”>

    <name>@pTableName</name>

    <value>’sys.Objects’</value>

                </schemaName>

                <schemaTable Include=”true”>

    <name>@pTableId</name>

    <value>342</value>

                </ schemaTable >

    </ItemGroup>

     

    The name element needs to be set to the name of the parameter including the @.  The value element needs to be set with the value to which the parameter has to be set to. The only gotach I have found so far is that string data needs to wrapped in the quotes. 

     

    Then to the get the MSBuild to pass the parameter into the Sql.Execute Task set a attribute on the task called Parameters and then @(<ItemName>).  The Parameters will take in a list of parameters.  Here is an example:  <Sql.Execute Path="CreateTable.sql" ServerName="$(SQLServer)" DatabaseName="$(Database)" Parameters="@( schemaName);@( schemaTable)" />

  • To Collate or not to Collate

    Collation plays an important role within the database, as it sets out how the SQL Server has to manage string data within storage & queries, yet I don’t script out the collation when scripting out my databases.  So why don’t I script out the collation if it so important? Its because if the collation is specified but not used properly within the database it can cause some of most common problems, like “Cannot resolve collation conflict for EQUAL TO operation” or “Cannot resolve collation conflict for UNION operation”. These errors are caused when string data within different collations are being compared or combined into the same result set; this is nothing new so why talk about collation now? Well it’s to do with the release of Visual Studio 2005 Team Edition for Database Professionals (VSTDB) something I think needed to be taken in consideration if this is being used to manage the database side of the project.

     

    A cause for theses types of problem is when a table is created on a database with the collation scripted out and deployed on another database which has a different collation to the create script.  With the VSTDB this will not occur as much because a project will be assigned a collation and the database will be created with this collation or if the database does not get recreated and has a different collation then the tables will be created with the project collation (This is the desired behavior when I saw this last and was confirmed when I spoke to the VSTDB Team in November). So I can see that VSTDB will help with reducing these types of errors.

     

    Another cause of the error is when creating temp tables without setting the collation to the same as the database collation and the server collation is different.  There are 2 ways to resolve this issue. The first is to rebuild your system tables to match the collation of the database that has been created or another way is by using the collation clause within the create temp table script(s) within the T-SQL code.  These errors could be become more frequent when using the VSTDB; because as mentioned before the database or tables that are create under the VSTDB are assigned the project collation, which will have to be of a type collation as there is no server default option and this could be different to that of the server that is being deployed to. 

     

    By taking the time to put in the extra collation clause in the temp table script(s) it reduce the risk of database code failing with “cannot resolve collation” errors on any SQL server configuration.  Also it reduce the temptation of people changing the VSTDB project collation to match there server collation if they can’t change their SQL server default collation and causing different behavior of SQL statements.

     

    I can understand the reason for making the database project ask for a collation because it means that when you unit test the code on any server it will always behave in the same way. Also it means as stated before, the database can run on any server with less risk of collation errors.

  • Delpoying Reporting Services Reports With MSBuild

    I am currently responsible for deploying our project into the test and production environments.  The project has a few reports which need to be installed on a few report servers.  The deployment tool that I am using for the rest of my project is MSBuild so I tried to see if there was a way of getting MSBuild to deploy the reports.  The only method that I found was to write an RS script file which deployed the reports and calls the Exec task to run the RS script. 

    So I started outright with RS script as I was not over joyed about writing VB.Net code as I am a C# coder. Then I remembered that MSBuild is customizable as it allows custom tasks to be written.  So I stopped writing my VB.Net code went back C# and created a few tasks that are commonly needed to deploy: AddReportUser, CreateReportFolder, CreateConnectionSoruces, DeployReports and SetReportsDataSource.  The task code was easy to write. I did have to change some of the auto generated code.  I changed the refence.cs file by changing the ReportingService2005 method to accept a string as it called and removed all references to the setting file and app.config.  I also deleted the setting file and app.config that was created by the Visual Studio.

      

    I have attached a copy of my report services MSBuild tasks with this post.  Also there is an example MSbuild file to show how to use these tasks.

  • Reporting Services Express Edition using host file redirect to connect to local database.

    I had a few reports for an application that needed to be deployed to number of different report servers.  The report servers were Express edition of Reporting Services 2005 with the application database on the same server which helped with the connection limitation of Reporting Services 2005 Express. As the report servers were joined to a domain their computer names have to be unique, the only way application was going to get around this problem was to use host file redirect.  To help with deploying the report I was to going to try to use the host idea, so I added the server name to the localhost entry and tried to run the reports.  Unfortunately this would not work because it looks like the reporting services 2005 express checks the name of the computer, which it gets from the following path HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\ComputerName\ActiveComputerName, against the datasource within the connection if they don’t match then raises an error.

  • Report Viewer Customizing Printing for Local Report

    We had a requirement to allow a front page, which was another local report, to be printed before the report within the report viewer was printed.  As both reports were local reports it was easy to create a print button which allowed us to meet the requirements, here is the code that we used:

    if (this.printFrontingSheetCheckbox.Checked)
    {
     LocalReport frontsheet = front.reportViewer1.LocalReport;
     Export(frontsheet, 8.27f, 11.69f, 0f, 0f, 0f, 0f);
     m_currentPageIndex = 0;
     Print(false);
     
     foreach (Stream s in m_streams)
     {
      s.Close();
     }
     front = null;
    }

    LocalReport report = reportViewer.LocalReport;
    Export(report, 11.69f, 8.27f, 0f, 0f, 0f, 0f);
    m_currentPageIndex = 0;

    Print(true); foreach (Stream s in m_streams)
    {
     s.Close();
    }

    private void Print(bool landscape)
    {
     if (m_streams == null || m_streams.Count == 0)
      return;
     PrintDocument printDoc = new PrintDocument();
     PrinterSettings ps = new PrinterSettings();
     ps.PrinterName = printDialog1.PrinterSettings.PrinterName;
     ps.DefaultPageSettings.Landscape = landscape;
        ps.DefaultPageSettings.PrinterSettings.DefaultPageSettings.Landscape = landscape;
     printDoc.PrinterSettings = ps;

     if (!printDoc.PrinterSettings.IsValid)
     {
      string msg = String.Format("Can't find printer \"{0}\".", printDialog1.PrinterSettings.PrinterName);
      MessageBox.Show(msg, "Print Error");
      return;
     }
     printDoc.PrintPage += new PrintPageEventHandler(PrintPage);
     printDoc.Print();
    } /// <summary>
    /// Export the given report as an EMF (Enhanced Metafile) file.
    /// </summary>
    private void Export(LocalReport report, float PageWidth, float PageHeight, float MarginTop, float MarginLeft, float MarginRight, float MarginBottom)
    {
     StringBuilder deviceInfosb = new StringBuilder();
     deviceInfosb.Append("<DeviceInfo>");
     deviceInfosb.Append("<OutputFormat>EMF</OutputFormat>");
     deviceInfosb.Append(string.Format("<PageWidth>{0}in</PageWidth>", PageWidth));
     deviceInfosb.Append(string.Format("<PageHeight>{0}in</PageHeight>", PageHeight));
     deviceInfosb.Append(string.Format("<MarginTop>{0}in</MarginTop>", MarginTop));
     deviceInfosb.Append(string.Format("<MarginLeft>{0}in</MarginLeft>", MarginLeft));
     deviceInfosb.Append(string.Format("<MarginRight>{0}in</MarginRight>", MarginRight));
     deviceInfosb.Append(string.Format("<MarginBottom>{0}in</MarginBottom>", MarginBottom));
     deviceInfosb.Append(string.Format("</DeviceInfo>"));
     string deviceInfo = deviceInfosb.ToString();
     Microsoft.Reporting.WinForms.Warning[] warnings;
     m_streams = new List<Stream>();
     report.Render("Image", deviceInfo, CreateStream, out warnings);
     foreach (Stream stream in m_streams)
     {
      stream.Position = 0;
     }
    }

    /// <summary>
    /// Handler for PrintPageEvents
    /// </summary>
    private void PrintPage(object sender, PrintPageEventArgs ev)
    {
     Metafile pageImage = new Metafile(m_streams[m_currentPageIndex]);
     ev.Graphics.DrawImage(pageImage, ev.PageBounds);
     m_currentPageIndex++;
     ev.HasMorePages = (m_currentPageIndex < m_streams.Count);
    }

    Then a new requirement came that we needed to get the report viewer print button to be the same as our print button.  At first I didn’t think it was possible because the button was within the report viewer control, but after looking through all the properties, methods and events we found the print event, which according to the object help, fires after the user prints a report.  This would help because at least we would be able to detect a print event and then print the front page at the end of printing the report.  But a collogue played around with the code and found that the event gets fired first before the printing happens and setting the cancel variable to true stops the report viewer doing the printing then by calling our printing code this allowed us to meet all the requirements.

  • Creating unique random number generator for SQL Server.

    I have worked on a project that required a random number, which had to be created within a certain range and be unique within the table that it’s going to be stored in.  The uniqueness of the number only needed to apply to numbers that were marked as active within the table.   Someone used a .Net approach whereas I tried using both SQL functions and  procedures both of which worked in the same manner.

    The method used within SQL Server would take a guid,(generated by the NewID) then strip out the numbers and this was then used as a starting point. If the number was not free it would then move up one and down one at a time to see if the number was free. This worked fine until the range of numbers free started to become limited. 

    The best method which is still within SQL server was to create a table with all the numbers within the range and mark them as free.  Then select the top 1 of all the numbers that are marked as free, this will allow the number to be unique, but order them by using the NewID function which will cause the number to come out in a random order.   Then take the number, mark it as in use until it becomes free again.

More Posts Next page »
Powered by Community Server (Personal Edition), by Telligent Systems