Welcome to blogs.conchango.com Sign in | Join | Help

Welcome to blogs.conchango.com

Steve Wright's Blog

  • Reporting Services 2008 Report automatic upgrade gotcha.

    For the Scrum for Team System project I have been focusing on the reports; I came across an issue which was due to the way the Reporting Services 2008 "automatic upgrade" feature upgraded our reports.

    The Scrum for Team System template contains over 20 reports all written for Reporting Services 2005. This was done because Team Foundation Server 2008 only worked on SQL Server 2005 and Reporting Services 2005. Then in August of 2008 Team Foundation Server 2008 Services Pack 1 was released which added support for the Team Foundation Server 2008 is to run on SQL Server 2008 platform.

    With Reporting Services 2008 there is a new reporting definition language schema. However reporting services does have backward compatibility method as describe below:

    The report server will validate report definition file against the included reference to the RDL namespace which specifies the version of the report definition schema that is used.

    The report is automatically upgraded the first time it is viewed, but the stored report definition file remains unchanged.

    So if you edit the report from the server it will still remain in old schema and not the new schema.

    To get automatically upgraded report definition you will open the report in one of the authoring tools: Business Intelligence Development Studio (BIDS) or Microsoft Report Builder 2.0.

    So I did some testing to see how the 2005 reports would look after the reporting services 2008 processing. I found that 2 reports didn’t render the same as did on reporting services 2005. The 2 reports had a common layout they were using a reporting services sleazy hack: The Green-Bar Matrix. The cell which controls the background colour was a lot more visible than the original as well as the text. So I rewrote the reports and were released them with version 2.2.

    Then one of our template users reported that one of the other reports stopped working. The report started to give the following error:

    The processing of Parent for the tablix ‘table1’ cannot be performed. The comparison failed. Please check the data type returned by the Parent.

    I was able to trace the error to an option in one of the report define groups: Recursive parent, which can found under advanced; removing the option made the report worked again. I haven’t been able trace why the error happens it only seems when our data came back in a certain way which I am currently looking into.

  • Some practical SQL Spatial tips.

    I have just finished a project were I made a lot of use of the SQL Spatial to do some processing and loading into the database. Here are some of lesson I learnt:

    • The first method that needs to call after instantiating a SQLGeographybuilder object is: SetSrid() then the BeginGeography(), BeginPoint().
    • Before using sending a SQLGeometry object to the SQL Server use the IsVaild() function to ensure that Geometry object is valid.

      I create a console application which loaded line, which was using OS coordinates system, data from a flat file. Some of the line data within the file wasn’t as correct as I was led to believe. The application create the SQLGeometry object .NET but the Sql Server then rasied the error when it received and try to save it into the table.

    • The STPointN() function is 1 base which is stated in the Books On-Line
    • If your query needs to select some of the derive data from the functions you can help performance by using persisted compute columns on the base table.

      I had to create a view which needed the latitude and longitude of the starting and end points of the geom line also including the distance. The view would take about 20 seconds to finish executing. Once I created some persisted compute columns on the base table the view would finish executing under 3 seconds

    • You can create more than one spatial index for a spatial column.
    • Trying to join tables using the spatial functions isn’t great for performance.
  • Working With Reporting Services Multiple Value Parameters

    A couple of colleagues of mine were working on a complicated reporting services project; they had one report which had a parameter which allowed a user to select multiple values of numbers, the problem they were facing was how pass the selected values into the query.

    They were looking for a way within SQL Server to split a string by a delimiter. As they were using a stored procedure to access the data and multiple value parameters, regardless of parameter type, are passed in as a string of comma separated values.

    One thing which they weren’t aware of was that with a multiple value parameter, the behaviour can differ between how the data is being retrieved i.e.: stored procedure or embed sql statement.

    If the data set is using direct query to gather the data then the multi value parameter can be used in the query with a IN clause like so:

    Where column1 in (@pMultiValueParam)

    What happens is that before the query is sent to the SQL Server the query the report server substitutes the @pMulitValueParam with a comma separated list which makes it valid in clause like so:

    Where column1 in (10, 3, 4, 56)

    Also it has been blogged (http://www.socha.com/blogs/john/2009/03/tfs-report-issues-with-sql-server-2008.html) that in reporting services 2008 there is a new change in the behaviour. You could see the following message when you have no values to select:

    Incorrect syntax near ')'.

    When there are no values to pass, Reporting Services 2008 simply removes @pMultiValueParam, so you get something like "Where column1 in ()" preventing this query from running.

    The solution is to add an expression to the pMultiValueParam report parameter. The expression is evaluated in order to determine what is passed to the query.

    Like so:

    =IIF(Parameters! pMultiValueParam.Count > 0, Parameters!pMultiValueParam.Value, "")

  • How to load spatial data into SQL Server 2008 from .Net

    I have been working on a project which made use of the spatial data type geography within SQL Server 2008.

    An issue that I had was how to load the geography data from a KML file into SQL Server 2008. Currently there is no out of the box tools to do this. There is a 3rd party tool, Safe FME, which offer either their own tool or components which extend integration services. This was overkill for my issues as I only had to do it once.

    So I wrote a console application which parses the KML file to extract the point data convert it to a SqlGeography type and store it in the database.

    To use the SqlGeography C# type you need to add reference to the following name space: Microsoft.SqlServer.Types. This can be found in the following dll: Microsoft.SqlServer.Types.dll

    Then use the following code to create the c# sqlgeography type:

    1:// use SqlGeographyBuilder to help create the SqlGeography type
    2: SqlGeographyBuilder geographyBuilder = new SqlGeographyBuilder();
    3: string[] longLat;
    4: SqlGeography geography;
    5:
    6: // gets the co-ordinates
    7: XElement coOrdinates = element.Element(ns + "Point").Element(ns + "coordinates");
    8:
    9: // set the Spatial Reference Identifiers that will used to create the point
    10: geographyBuilder.SetSrid(4326);
    11:
    12: // state what type of geography object that I to create
    13: geographyBuilder.BeginGeography(OpenGisGeographyType.Point);
    14:
    15: longLat = coOrdinates.Value.Split(new char[1] {','});
    16:
    17: // add the frist figure lat long point
    18: geographyBuilder.BeginFigure(Convert.ToDouble(longLat[1]), Convert.ToDouble(longLat[0]));
    19:
    20: // close the figure and geography class
    21: geographyBuilder.EndFigure();
    22: geographyBuilder.EndGeography();
    23:
    24: // get the geography builder to return the sqlgeography type
    25: geography = geographyBuilder.ConstructedGeography;
    26:
    27: return geography;

    After creating the data type I then needed to write the code send the data to the SQL server which as follows:

    1: // set the command text
    2: string sqlCommandText = "insert into [dbo].[Location]([Location],[CoOrdinates]) Values(@pLocation,@pCoOrdinates)";
    3:
    4: // create the command object and set which command type
    5: SqlCommand sqlCommand = new SqlCommand(sqlCommandText, sqlConnection);
    6: sqlCommand.CommandType = CommandType.Text;
    7:
    8: // create and add the paramter for standard sql data type
    9: sqlCommand.Parameters.Add(new SqlParameter("@pLocation", name.Value));
    10:
    11: // create and add the paramter for sql geography data type
    12: // as I am using and system CLR type have to say what data type name the parameter is
    13: sqlCommand.Parameters.Add(new SqlParameter("@pCoOrdinates", geography) {UdtTypeName = "Geography"});
    14:
    15: // execute the command
    16: sqlCommand.ExecuteNonQuery();

    Like the report viewer control the Microsoft.SqlServer.Types namespace is not installed with .Net. For the application to work on another computer, without having to install SQL server or their client tools, a Redistributable package would need to be installed.

    The SQL Server System CLR Types package contains the components implementing the new geometry, geography, and hierarchyid types in SQL Server 2008. This component can be installed separately from the server to allow client applications to use these types outside of the server.

    A version can be found here:

    http://www.microsoft.com/downloads/details.aspx?FamilyId=C6C3E9EF-BA29-4A43-8D69-A2BED18FE73C&displaylang=en

  • Business Intelligence with SQL Server 2008 Geometry Data Type.

    I am currently working on a project which is automating a business intelligence process base upon images and image recognition. The process is as follows:

    • Take a photography
    • Put the photography through the image recognition software and record the recognition data.
    • Process the recognition data into meaningful business metrics
    • Produce a report on metrics

    The part of the process, from above, that I am helping with is processing the recognition data (the hit data) into meaningful business data. This process simply takes the hit data (which is a point), then creates some square blocks, which I use to group the hits data and perform some metric operations. The metrics currently use data about the area and the density of hits within the area. To help process this data I decided to make use of the SQL Server 2008 spatial type: Geometry.

    Firstly I would like to point out that I didn’t have to use the geometry type. As I am currently dealing with square areas, I can group or locate all the hits that fall within area by using the following filter clause: HitY between MinY and MaxY and HitX between MinX and MaxX. However by reading the filter alone can cause misunderstanding on what the filter is suppose to be doing. As I am using the geometry data type that filter is replaced by something which does state what the filter is doing, like so: area.STIntersects(HitPoint) = 1 with the added bonus of less code being written and read as well as less chance of a bug being created.

    Another point I would to make is that by using the geometry type my code is adaptable to changes. Currently our process is working to square areas and in the future that might change. Using the standard SQL filters that would mean all the filter clauses of the data would have to be updated. Also I would need to be more sophisticated, which could take a longer development and testing cycle which could increase the risk of issues. With the use of the geometry type, only the way that the area polygon is created would need to be updated which will help in reducing the development and testing cycle which interns reduces the risk of issues.

    Another benefit of having some of the data stored as the geometry spatial data type is that I was able to evaluate or debug the steps of the process far easier. This is due to the way that SQL Server Management Studio display spatial data as shown below.

    SpatialGrid

  • Scrum for Team System Version 2.2 support for SQL Server 2008

    As I have mention in a previous blog entry I’ve been heavily involved in the Scrum for Team System process template since the start of version 2 and have written the majority of the reports. I have faced, and still do, some challenges while doing this. Some of them have been around how to display the data within reporting services 2005.

    Now with the release of Team Foundation Server 2008 Service Pack 1, release in August, this enables Team Foundation Server 2008 to be used on a SQL Server 2008 environment. This will make some of reporting challenges I faced with reporting services 2005 disappear. This will due to major changes that been made to reporting services 2008 like Tablix control and wider Dundas chart controls.

    Within this release we have made a patch which will update some of the reports for reporting services 2008. This to fix some of the reporting services 2005 Patterns and Recipes which became visible. I am currently working looking at how Scrum for Team System can make more use of the reporting services 2008 which we will release as an advance reporting pack in the future.

    To find out more about the Scrum for Team System SQL Server 2008 patcher please read the following Scrum for Team System blog

    New Reporting Features available in Scrum for Team System v2.2

    Build Reports added to Scrum for Team System v2.2

    The Scrum for Team System report slide show tool

    Scrum for Team System Version 2.2 support for SQL Server 2008

  • The Scrum for Team System report slide show tool

    As I have mention in another blog posts; I’ve been heavily involved in the Scrum for Team System process template since the start of version 2 and to coincide with the release of version 2.2; I would like bring your attention to a new tool, which I help to developed, that has been added to this release.

    I started creating this tool due to some feedback we got from our beta users for “TaskBoard for Team system” tool. The request was to add a demonstration mode into the application which would show things like swimming lane-view and the following reports: Sprint burndown chart; Sprint cumulative flow as these can be view within the application.

    The reason for the request was create the sense of: "Look guys, this is what we are working on, and this is how we are doing", throughout the day, not only during the scrum meetings. This also happen a lot within our projects which they do something similar as well. Another reason for the tool to separate from the Task Board application was to avoid polluting what the Task Board application was suppose to do.

    So a new page was added to ScrumforTeamSystem virtual folder called: ReportSlideShow. This will allow a selection of team system project(s) reports to be display by their parameters default values. However the reports don’t have to be scrum for team system only reports there is one caveat: all reports have to exist in all requested display projects.

    The report that are which are display is control by configuration file. To see how to figure which reports are display please use the Report Slide User Guide Post from the Scrum for Team System site

    New Reporting Features available in Scrum for Team System v2.2

    Build Reports added to Scrum for Team System v2.2

    The Scrum for Team System report slide show tool

    Scrum for Team System Version 2.2 support for SQL Server 2008

  • Build Reports added to Scrum for Team System v2.2

    Continuing from my previous posted. I’d like to give an overview to the new engineering practices reports included in the Release 2.2.

    Builds
    Across a user selected time period (defaulting to the current sprint), this report shows the total builds per day broken down by status.

    Build Static Analysis and Compile
    This displays the number of Errors and Warnings either compile or static analysis the last N number of builds.

    Compile

    Build Unit Tests
    Shows the total number of units test run from the last N number of builds. The total is broken down by number status of unit tests for each build.

    UnitTest

    Last Build Unit Test Results
    This report displays the list of each test that was run for the last build and shows the result of that build. If the unit test failed it will also display the fail message as well. The last build is taken to be the last build within the data warehouse.

    lastunittest

    Code Coverage
    This report shows the code coverage of and the total of code churn for last N number of builds. The coverage units can be selected between blocks, which is the visual studio default, lines or partial lines.

    code coverage

    Build Code Churn
    This report shows the total “Code Churn Counts” of the last N number of builds broken down by Added, Modified and Deleted lines. This report also features a drill down option which links to the Build Files reports.

    CodeChurn

    Build Files
    This report, which is also a drill down for the Build Code Churn report, will display the files which were in the build or between two builds. The files are grouped by the folder in which they are located within source control and display the number of Added, Modified and Deleted lines for each of the change sets. Also included is the following change set information: Check-in by, Date, Policy Override Comment.

    BuildFiles

    Build Quality
    This report is a based on the MSF for Agile Software Development Quality Indicators. This has been added by popular request and inspired by a blog post by Ed Blankenship: (http://blogs.infragistics.com/blogs/eblankenship/archive/2007/06/18/msf-agile-quality-indicators-report-for-conchango-scrum-process-template.aspx).

    Build Quality

    Build Duration
    This shows how long the last N builds took to complete.

    Duration

    Build League
    This will show a League table of the Top N number of people on their total number builds of Fail or Successful Builds (using either an absolute number or percentage base). This report will run for a selected period defaulting to the currently running sprint. A successful build is defined as any build which not in a state of failed. (Which is not the same as a successful build.)

    League

    There is one caveat: The way in which a build is counted against a person is if their change set is linked to a build. So some people will gain some builds fails even if their change set is actually not the reason why the build failed.

    All the reports will have the option to select which build type to run against. Also where the reports allow an N number of last builds they have been limited to the following values 10,15,20,25 or 30.

    These reports are not actually link to the Scrum methodology but are helpful tools in the engineering practices. There will be a separate forum for issues and question which can be found here: Engineering Practices – v2.2 Reports 

    New Reporting Features available in Scrum for Team System v2.2

    Build Reports added to Scrum for Team System v2.2

    The Scrum for Team System report slide show tool

    Scrum for Team System Version 2.2 support for SQL Server 2008

  • New Reporting Features available in Scrum for Team System v2.2

    To coincide with the release of version 2.2 of Scrum for Team System, I’d like to bring your attention to the wide range of new reports included in this release. I’ve been heavily involved in the Scrum for Team System process template since the start of version 2 and have written the majority of the new reports and tweaked some of the old ones to improve performance by moving some of the querying to the TFS cube, as well as reflecting user feedback.

    One such fix, which can be found in release, was to the burn down report which would show a misleading (but accurate) trend line if the task data wasn’t updated on the very first day of the sprint. (This could also be fixed by making your Scrum Master work harder, but I don’t have any control over that so I decided to tweak the report instead).

    Most of the new reports in this release are focussed on engineering practices, which I will cover another blog post; however there is one new report specifically targeted for Scrum called the velocity report.

    Velocity is based upon the recorded historical performance of a team, and is a fantastic tool for accurate release planning; for more details please see my colleague Simon Bennett’s blog for more details.

    Velocity

    So onto the new reports....

    New Reporting Features available in Scrum for Team System v2.2

    Build Reports added to Scrum for Team System v2.2

    The Scrum for Team System report slide show tool

    Scrum for Team System Version 2.2 support for SQL Server 2008

  • Using NDepend to help guide Refactoring

    In my other blogs entries I mention that I have been looking into building a Team Foundation Server Data Warehouse Adapter.  After I got my initial proof of concept version working, I started to extend it to get a list of available builds from TFS and then import the output of Source Monitor – a popular free code metrics tool.

    Now I must state that I am not a natural .NET coder.  My normal day-to-day work is based around the SQL Server technology stack. I do have some .NET coding skills but generally just enough for what is need in and around SQL Server.  So I asked my colleague, Howard van Rooijen who is one of our top .NET coders, to have a look at my code and give me some hints.

    This provide me with a great opportunity to learn from Howard about some more advance Design Patterns,  Visual Studio tips and tricks, and how to effectively use some tools I’ve not used before to help me write better code. The first thing Howard did was install the following on my machine:

    • ReSharper – to help me refactor the code, fortunately we have just rolled out ReSharper licenses to all .NET Developers!
    • StyleCop – to help remind me of the Microsoft C# Coding Conventions
    • TDD.NET  to ease the pain of running my unit tests
    • NDepend – a code metrics tool

    Howard wanted to integrate NDepend into my TFS Data Warehouse Adapter and suggested that the best way to understand the tool would be to use it to analyse my TFS Adapter code and see if it would be able to give the direction on where to focus our refactoring efforts.   

    NDepend  comes with 82 code metrics out of the box, but it’s far more than a standard code metrics tool. It comes with a bundle of best practice design rules which are written using Code Query Language – an idea so simple and elegant that I bet the FxCop team are kicking themselves for not thinking of it first. CQL allows you to write statements like:

    // <Name>Methods poorly commented</Name>
    WARN IF Count > 0 IN SELECT METHODS WHERE PercentageComment < 20 AND NbLinesOfCode > 15  ORDER BY PercentageComment ASC

    You can even embed CQL directly into your code via attributes. NDepend also has features like being able to compare different builds (Patrick Smacchia, NDepend’s developer demonstrated this feature on the recent .NET Framework 3.5 SP1 release), manage complexity, dependencies and monitor the health of your build process.

    We started by running my code through NDepend and the initial results were that my code has high instability and not at all abstracted.  I couldn’t really affect the instability as my code was just a thin wrapper around the TFS API.  It also highlight that some of my old best practices that needed updating as many of them are now redundant with C# 2.0 features (generics, static classes etc). It also highlighted performance gains I could make.

    Howard said that he was having some difficulties understanding what the intention of my code was, as he was unfamiliar with building Data Warehouse Adapters. He asked me if I could write some unit tests that would call the code and verify the output – so he could use the “Test with... Debugger” feature of TDD.NET to see what was going on with the code.

    Once I had written some tests, Howard wrote two more: one that passed in a null value and another that passed in an empty string; these, he explained were two valid edge cases for handling bad data and to my surprise both tests failed! Once I fixed both failing tests, he asked me to run the whole test suite again and this time I was really surprised as all of my previously passing tests now failed! The issue was with two helper methods that convert string based representations of number within the Data Warehouse into valid number primitives.

    NDepend pointed out lots of issues with these methods and Howard said that a simple Regular Expression could replace a lot of the parsing logic I had written, so I set about refactoring the code.

    Before:

    public static string CleanInt(string number)
    {

        string result = string.Empty; 
        bool foundNonDigit = false
        int index = 0;

        if (number == string.Empty) 
       
            return "0";
        }

        while (!foundNonDigit) 
       
            if (char.IsDigit(number[index])) 
           
                result = string.Format("{0}{1}", result, number[index]); 
                index++; 
           
            else 
           
                foundNonDigit = result.Length > 0; 
           
       
        return result;
    }

    public static string CleanFloat(string number)

        string result = string.Empty; 
        bool foundNonDigit = false
        bool foundDecmialPoint = false
        int index = 0;

        if (number == string.Empty) 
       
            return "0"
       
        while (!foundNonDigit) 
       
            if (char.IsDigit(number[index]) || (number[index] == '.' && !foundDecmialPoint)) 
           
                result = string.Format("{0}{1}", result, number[index]); 
                foundDecmialPoint = number[index] == '.'
                index++; 
           
            else 
           
                foundNonDigit = result.Length > 0; 
           
       
        return result;
    }

    After:

    public static string CleanInt(string number)

        int resultFromTryParse; 
        // check for empty string after cleaning string 
        if (number.Trim() == string.Empty) 
       
            return "0"
       
        // if try parse works return the number in string format 
        if (int.TryParse(number, out resultFromTryParse)) 
       
            return resultFromTryParse.ToString(); 
        }

        // else return results of clean GetCleanNumberString 
        return GetCleanNumberString(number, "Integers");
    }

    public static string CleanFloat(string number)

        float resultFromTryParse; 
        // clean the number and check to see if it's empty 
        number = number.Trim(); 
        if (number == string.Empty) 
       
            return "0"
       
        // try parse the number if works the return the number 
        if (float.TryParse(number, out resultFromTryParse)) 
       
            return number; 
       
        // returns result of Get clean number 
        return GetCleanNumberString(number, "Floats");
    }

    private static string GetCleanNumberString(string number, string numberType)

        // Create a regualr patten with name groups do a macth on the number entered 
        Regex regex = new Regex(@"(?<Floats>(?<Integers>\d+)+\.\d+)|(?<Integers>\d+)", RegexOptions.Singleline); 
        Match match = regex.Match(number); 
        string results = string.Empty;  

        // if match groups are found the return the string 
        if (match.Groups != null
       
            results = match.Groups[numberType].Value; 
       
        return results;
    }

    Now all my tests passed and NDepend was happy too.

    Next NDepend highlighted some of my methods that had too many lines of code. Howard said he was having a bit of difficulty understanding what the code was doing and explained to me the notion of Programming by Intention.

    Next he paired with me and went through the method reformatting the code. He said that by reformatting, reorganising and removing redundant variables you start to see patterns forming within the code, that clumps of logic naturally come together and how you can then use ReSharper’s Extract Method refactoring to extract that clump of code into a separate method. Now instead of having a mass of hard to understand code – the method simply listed a series of steps that occurred – the intention of the method had been made clear. Howard was happy because he now understood what the method was doing and NDepend was happy as the method was now shorter and a great deal less complex.

    In my original proof of concept I had used partial classes as a mechanism to extend the main TFS Data Warehouse Adapter and implemented another partial class in order to integrate Source Monitor.  Howard pointed out that this approach wouldn’t work if we wanted to integrate a second tool.

    He said that now we had cleaned up the code and broken down the few large methods into a series of smaller methods we had surfaced the behaviour required to integrate an external code metrics tool into the Data Warehouse Adapter and that we could use ReSharper’s Extract Interface refactoring to move the definitions of this behaviour into an interface called ICodeMetricAdapter and then use this interface to implement any other 3rd party code metrics tool.

    Howard said that we had now essentially implemented the adapter pattern. Next we refactored the existing code to use the interface instead of the concrete type and we extracted the Source Monitor code into a new file called SourceMonitorAdapter which implemented the ICodeMetricAdapter interface. We added a List<ICodeMetricAdapter> to the parent object and then refactored the code to loop through this list and then call the current ICodeMetricAdapter.

    We managed to get this all done in one afternoon, just as I was packing up Howard asked me “Now, what do you need to do to add an NDepend Adapter?” I created a new class called NDependAdapter and made it implement ICodeMetricAdapter. Howard showed me that I could use the Implement Members quick fix by selecting the interface and hitting ALT-ENTER. Then all I had to do was add a new instance of the NDependAdapter to the List<ICodeMetricAdapter> and the new NDependAdapter is automatically called!

    Below is the before and after class diagrams of the solution:

    Before:

     

    before

    After:

    After

     

  • How I built a Team Foundation Server custom data warehouse adapter

    As mentioned in a previous blog I would like to explain some of the steps I took to get my custom data warehouse adapter to work.  I am going to start from the beginning just so that I have an easy starting point.  The code I show here was based upon my POC and written in C#.

    1. Create a new C# Class Library project in Visual Studio. Then add references at least to the following assemblies: Microsoft.TeamFoundation.dll, Microsoft.TeamFoundation.Client.dll, Microsoft.TeamFoundation.Common.dll, Microsoft.TeamFoundation.Warehouse.dll and System.Web. As I need to connect to the build server I added Microsoft.TeamFoundation.Build.Common.dll and Microsoft.TeamFoundation.Build.Client.dll
    2. Then specify the use of the IWarehouseAdapter interface.
    3. Implement IWarehouseAdapter.RequestStop which only sets a stop flag to true which can then be tested by the other methods periodically.
    4. Implement the IWarehouseAdapter.Initialize. This method this where I need to store objects that will need to communicate to the all systems that you adapter needs to talk to i.e.:
    string url;
    TeamFoundationServer tfs;
    _DataStore = ds;
    if (_DataStore == null)
    {
       throw new Exception("Null data store.");
    }
    url = Microsoft.TeamFoundation.Server.TeamFoundationApplication.TfsNameUrl;
    tfs = TeamFoundationServerFactory.GetServer(url);
    if (tfs == null)
    {
       throw new Exception("TF Server instance not obtained for TFS url: " + url);
    }
    _BuildServer = (IBuildServer)tfs.GetService(typeof(IBuildServer));
    if (_BuildServer == null)
    {
       throw new Exception("Build Server instance not obtained for TFS url: " + url);
    }
    _CommonStructureService = (ICommonStructureService)tfs.GetService(typeof(ICommonStructureService));
    if (_CommonStructureService == null)
    {
       throw new Exception("Common Structure Service instance not obtained for TFS url: " + url);
    }
    5. Implement the IWarehouseAdapter.MakeSchemaChanges. This method is called to see if there are schema changes that would need to be made to the data warehouse.  This is recommended to be where you actually register your changes you need to make to the warehouse i.e.: 
    SchemaChangesResult result = SchemaChangesResult.NoChanges;

    //Get the current data warehouse configuration 

    WarehouseConfig warehouseConfig = _DataStore.GetWarehouseConfig();

    //As the IWarehouseAdapter.MakeSchemaChanges can be called many times in one sessions
    //I needed to see if your fact was already created

    Fact MyNewFact = warehouseConfig.GetFact("MyNewFact");
    if (MyNewFact == null)
    {

    //Create My New Fact

    MyNewFact = new Fact();
      MyNewFact.Name = "MyNewFact";

    //Adding a link to the Team Project – I had to do this otherwise my save would not work

    MyNewFact.DimensionUses.Add(CreateDimensionUse("Team Project", "Team Project"));

    //Adding a link to the Build dimension also when use the dimension I needed to
    //find out the dimension key attribute was for when saving my data.

    MyNewFact.DimensionUses.Add(CreateDimensionUse("Build", "Build"));
    MyNewFact.Fields.Add(CreateField("measure1","int",0,"Sum"));
    MyNewFact.Fields.Add(CreateField("measure2", "int",  0, "Sum"));

    //Setting which perspective it would show under if enterprise edition of AS was in use.

     MyNewFact.PerspectiveName = "Code Churn";
     MyNewFact.IncludeCountMeasure = true;
    if (!_StopRequest)
    {

    //Starting a Transaction to rollback my changes if they fail

     _DataStore.BeginTransaction();
    try
    {

    //Adding to the data warehouse configuration xml file and saving the changes

    warehouseConfig.Facts.Add(MyNewFact);
     _DataStore.Add(warehouseConfig);
     _DataStore.CommitTransaction();
     result = SchemaChangesResult.ChangesComplete;

    }
    catch
    {
    _DataStore.RollbackTransaction();
    throw;
    }
    }
    else
    {
    result = SchemaChangesResult.StopRequested;
    }

    }
    return result;

    //The functions to create a new field and dimension link

    private Field CreateField(string FieldName, string FieldType, short FieldLength, string FieldAggregationFunction)
    {

    Field newField = new Field();
    newField.Name = FieldName;
    newField.Type = FieldType;
    newField.Length = FieldLength;
    newField.AggregationFunction = FieldAggregationFunction;
    return newField;

    }

    private DimensionUse CreateDimensionUse(string DimensionName, string UseName)
    {

    DimensionUse dimensionUse = new DimensionUse();
    dimensionUse.DimensionName = DimensionName;
    dimensionUse.UseName = UseName;
    return dimensionUse;

    }

    6. IWarehouseAdapter.MakeDataChanges which is where all the transferring and transforming of the data takes place.

    IEnumerator projectEnum = _CommonStructureService.ListAllProjects().GetEnumerator();
    DataChangesResult result = DataChangesResult.NoChanges;
    IEnumerator buildEnum;
    IBuildDetail[] buildDetails;
    ProjectInfo currentTeamProject;
    DateTime buildStartSearch = LastMyNewFactPrcoessedDateTime;
    DateTime buildEndSearch = DateTime.Now; while (projectEnum.MoveNext())
    {

    //checking to see if a stop request has happend if so then return the fact that a stop happen.

    if (_StopRequest
    {

    return DataChangesResult.StopRequested;
    }

    //Just checking the state of the project as I am not interested in deleted projects

    currentTeamProject = projectEnum.Current as ProjectInfo;
    if (currentTeamProject.Status != ProjectState.Deleting)
    {

    //Get a list of builds and the filter the out builds that I have already processed.

    buildDetails = _BuildServer.QueryBuilds(currentTeamProject.Name);
    var filterBuildList = from bd in buildDetails
    where bd.StartTime >= buildStartSearch
    && bd.FinishTime < buildEndSearch
    select bd;

    //From my new list of builds save my fact details

    buildEnum = filterBuildList.GetEnumerator();

    while (buildEnum.MoveNext())
    {

    if (_StopRequest)
    {
    return DataChangesResult.StopRequested;

    }
    SaveMyNewFactEntry (buildEnum.Current as IBuildDetail, currentTeamProject);

    //Update the last build started and date I have just covered.

    LastMyNewFactPrcoessedDateTime = buildEndSearch;
    result = DataChangesResult.ChangesComplete;

    }
    }

    }

    private void SaveMyNewFactEntry (IBuildDetail buildDetail, ProjectInfo currentProject)
    {

    Random rand = new Random();

    //Check to see if this build has been already added to fact table.

    string buildURI = LinkingUtilities.DecodeUri(buildDetail.Uri.AbsoluteUri).ToolSpecificId;
    if (_DataStore.GetFactEntry("Build Details", buildURI) != null)
    {


    FactEntry newFactEntry = _DataStore.CreateFactEntry("MyNewFact ");

    // Had to add Tracking ID by not doing so cause an error.  The Tracking ID is used to find facts
    //So need make sure that method which repeatable for the same fact entry each time.

    newFactEntry.TrackingId = Guid.NewGuid().ToString("D", CultureInfo.InvariantCulture);
    newFactEntry ["Team Project"] = LinkingUtilities.DecodeUri(currentProject.Uri).ToolSpecificId;
    newFactEntry ["Build"] = buildURI;
    newFactEntry ["measure1"] = rand.Next(0,100);
    newFactEntry ["measure2"] = rand.Next(0, 100);

    //Again create a transaction save the entry if successful commit the changes

    _DataStore.BeginTransaction();
    try
    {

    _DataStore.SaveFactEntry(newFact, true);
    _DataStore.CommitTransaction();
    }
    catch
    {

    _DataStore.RollbackTransaction();
    throw;
    }
    }
    }

    7. I need to store the last date processed so I created a property which stores the data in the datastore property bag.

    // Last changeset that was populated.

    private DateTime LastMyNewFactPrcoessedDateTime
    {

    get
    {
    String LastMyNewFactPrcoessedDateTimeStr = _DataStore.GetProperty("Last MyNewFact Prcoessed DateTime");
    DateTime lastMyNewFactPrcoessedDateTime = DateTime.MinValue;
    if (!String.IsNullOrEmpty(LastMyNewFactPrcoessedDateTimeStr))
    {
    lastMyNewFactPrcoessedDateTime = DateTime.Parse(LastMyNewFactPrcoessedDateTimeStr);
    }
    return lastMyNewFactPrcoessedDateTime;
    }

    set
    {

    _DataStore.BeginTransaction();
    try
    {
    _DataStore.SetProperty("Last MyNewFact Prcoessed DateTime", value.ToString());
    _DataStore.CommitTransaction();
    }
    catch
    {
    _DataStore.RollbackTransaction();
    throw;
    }
    }
    }


    8. Build the adapter as a DLL.

    9. Copy the build DLL into the warehouse plugins folder on the application tier.  In most cases will be C:\Program Files\Microsoft Visual Studio 2008 Team Foundation Server\Web Services\Warehouse\bin\Plugins.

    10. Reset IIS.

    11. On the application tier, navigate to http://localhost:8080/Warehouse/v1.0/warehousecontroller.asmx?op=Run and click Invoke.

    12. On the application tier, navigate to http://localhost:8080/Warehouse/v1.0/warehousecontroller.asmx?op=GetWarehouseStatus and click Invoke. Continue doing this until the status is returned as Idle. Check the application event log, to see if an exception has been thrown.

    13. Reset IIS.

    If you wish to find out what your current warehouse configuration is then run the following query against your TFS warehouse:

    SELECT CAST(wc.Setting AS XML) AS setting FROM dbo.[_WarehouseConfig] AS wc WHERE wc.ID = 'ConfigXML'

  • Building a Team Foundation Server custom data warehouse adapter

    I have been investigating a way of adding more metrics about our builds into the Team Foundation Server 2008 warehouse.  This would then allow us to measure quality of our code over time by other metrics than the just method offered by the Team Foundation Server 2008.

    So my goal was to create a new fact table and link this to build dimension.  The reason for building a fact table was the metrics as I would want to do some aggregation against them; this would mean I need to treat them as measures and measures only live in a fact table.  I also wanted a separate fact table so my adapter doesn’t impact the standard TFS adapters. 
    As my facts would be generated by builds it would mean that facts would link to the build which why I used the build dimension.  Also I found out later I had to also include the Team Project dimension as well.

    I managed to find some useful links that helped me write my adapter:

    http://msdn.microsoft.com/en-us/library/bb130342.aspx - This section covers several topics: how to implement an adapter; how the data warehouse is created and gives example on how to create adapter.  The example is only how to extend existing dimensions not how to create new objects.

    http://tomfury.spaces.live.com/blog/ - This guy was trying to do the same thing as me and created a series of blog entries explaining what he was doing and his issues he was facing and gave some code examples.  Also he has posted the following questions on the forums which were helpful:

    http://forums.microsoft.com/MSDN/showpost.aspx?postid=2464644&siteid=1
    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=1644099&SiteID=1
    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2518423&SiteID=1

    There some has been some changes, as I understand, between 2005 and 2008. As I no longer code against TFS 2005 I can’t make any real comment about these.  However I found these links helpful for coding against TFS 2008:

    http://ozgrant.com/2006/06/18/how-to-list-tfs-team-projects-using-the-api/ - this to gain information about the projects that are in TFS server.

    http://notsosmartbuilder.blogspot.com/2007/09/team-build-2008-api.html - this helped me to query TFS server about the builds it understands.  As I can see there has been a lot of changes between the 2005 and 2008.  In the 2008 API some namespaces have been made obsolete (Microsoft.TeamFoundation.Build.Proxy; also a few classes in Microsoft.TeamFoundation.Build.Common).

    To see how I wrote my custom TFS data warehouse adapter please click here.

  • Could HTTP Soap / T-SQL endpoint be replaced by Reporting Services XML Reports?

    I have been working on a POC using SQL Server 2008 and Google Maps.  The POC was to use Geo Spatial technology to illustrate our Geo Spatial data easily.   I will talk about this in more detail in another blog post as I would like talk about a problem I faced while creating this POC. 

    The problem I faced was how could I get my Geo Spatial data, which was stored within SQL Server 2008 using the new geography data type, into  "static" HTML?

    So I thought that SQL Server HTTP Endpoints for SQL would be great thing to use to solve this problem.   As all I was after was xml document to stream over HTTP.   There was no business logic needed so all the .Net would be is just another layer acting as go between.  Also I didn’t what to waste time on what code which wouldn’t have added any value.

    So I created my stored procedure to produce an XML document that I found usable by using the For XML Path.  Then I started to look at how to create a HTTP endpoint for SOAP. After working through this I was presented with this message from my SQL Server 2008: "Creating and altering SOAP endpoints will be removed in a future version of SQL Server. Avoid using this feature in new development work, and plan to modify applications that currently use it."

    It was nice to see that Microsoft are giving people plenty or warning about breaking changes for next version of SQL server.  It’s a surprise that a feature is being dropped so quick, second only to Notification Services since I have known the product, but I can understand the decision as a lot of questions on how, why  &  should it be used were ask by some of our SQL Server developers.   So I took notice of Microsoft warning and tried to see if there was another method I could get my data into an XML document and streamed over HTTP without using .Net code and web services.

    All I was doing with this data was just reading for display via another component.  The data was in a structured format.   Then I thought about Reporting Services, as you can write a report and get the reporting server to render to XML.   So I quickly wrote a report to get the outputted xml into the format that I wanted.  Then in my static html I changed my http request to point to my report server with added option to render to XML.

    This approach did work and was great, as I managed to produce a page that could request data XML data and render data without any need to write .Net code.  The static html page would always remain up to date.  Another bonus was I had the report ready to show the data in a table from without having to re-write or reduplicating SQL code.  I was also then able to extend this to accept parameters so I could order the data base upon some user data from the HTML input control.

    Now this approach is not always going to work for every situation.  One reason is that reporting services security will not allow anonymous access.  So when using a windows security the report server would need to be on the same server.  A method to overcome this would be to create some forms authentication method on the report server and get your site to create the form cookie automatically. 

    Another potential problem is way that Reporting Services sends the XML data other HTTP.  The XML data is actually sent as an attachment file on the HTTP response.  Now some applications, like the side bar gadgets, might not be allowed to, or can’t, handle http attachments.

  • How E-mail the Sprint Burndown Chart to yourself or your team.

    I have been working on Scrum for Team System focusing on the reporting functionality of this tool.  One of the things I looked at was to get the reports delivered to me by e-mail.  I went looking to see if there was an interface via the Team Explorer interface, to which I found quickly there wasn’t.  However because TFS uses Reporting Services I knew that I could use Reporting Services subscriptions functionality. 

    Firstly, if not already configured, the Reporting Services server will need to will need to be set up to point to an SMTP server.  This can be done by using the Reporting Services configuration tool and updating the e-mail settings within this tool.  

    You can find this tool on your TFS server under the following: Start Menu > All Programs > Microsoft SQL Server 2005 > Configuration Tools

    To create the subscription you need to use the Report Manager  (to gain access to this you can right mouse click on the report folder within the team explorer):

    When the report manager is loaded then the root folder will contains folders based upon the name of Team  Projects. Click on the folder that contains the required reports. After the list of reports have loaded click on the Sprint Burndown report to view the report.  After the report has rendered select the subscriptions tab, once the page has loaded and then click on new Subscriptions.  Then filling the necessary details on how the report should be delivered and what report parameters the report should used when it runs.  Finally I would recommend that for the parameters select the option to pick the defaults of the reports.

  • How to tell when the Team Foundation Server data warehouse was last updated.

    While I have been working on the Scrum for Team System reporting I came across a re-occurring issue.  The issue was not knowing when our data warehouse last updated or when the next update was due to happen. 

    After some investigation, including some information from a link on one of my colleague’s blog posts, I found the answer to the issue.  In the TFS  data warehouse there is a table called: _WarehouseConfig.  This contains some useful information like when the last data warehouse was last process time, process interval and end time of each step of the process ended.  

    So with this information I updated the version report for Scrum for Team System to include the following information: process interval time, next time process is due, last time the cube was processed, last time the data in data warehouse was updated.

    I have attached a copy of the new report as it not exclusive to Scrum for Team System.  Also this report was written for TFS 2008.

Powered by Community Server (Personal Edition), by Telligent Systems