Using Specflow and Visual Studio to generate test summary report

Okay, so you've been using Specflow for your Acceptance Tests, you have started to build up your tests suites for different components of your System. Now, let's say you want to somehow consolidate your results in a report to show your boss, manager or just to show off to the guy sitting next to you....

You could run your tests individually and see the results in Visual Studio Test Explorer, or maybe use what Marcus did in his blog. Or maybe, you can create a Windows Form application that will customize and simplify the process for you, just make a few inputs and you're good...you have a pretty HTML report that you can show your co-workers.

Speflow reporting anyone?

I was in the same predicament when our company started embracing Agile as part of our Agile Transformation process. I recommended to them a headless type of testing framework that did not rely on mouse clicks, and screen element validation to do some sort of UI testing, which can sometimes be unreliable and a bit slow. Now, as we started to accumulate tests and feature files, I started looking at ways we can somehow report these tests in a format that can be read and easily understood by anybody in the organization.

Marcus had a great article that I came across and I started to play around with this. Although his approach was to create an external tool that can be accessed via Visual Studio. I wanted to extend this to other folks, like SDETs in our organization and also added a few options so you can parameterize inputs that goes into your report. That's how I came up with the solution/project that I'm sharing with you. If you just want the solution and dig in right through the code, send me an email and I'll forward you the complete solution plus the test project.

Building the Solution

Okay, lets get this started. First off, you need to have a Test Project where you have your Specflow tests wired up. For practice purposes, I just created a simple test project and had 4 scenarios wired up with Specflow as below:

Fig 1

As you can see from the screenshot above, this example is pretty straightforward. I have defined 4 scenarios, the first 2 are for adding two numbers, the 3rd test is for subtraction and the 4th test is for multiplying 2 numbers. Notice that I also put in some custom tags - positivetest, negativetest, notimplementedtest. I'll get back to that in a sec. Even more straightforward is the test execution step definition file for the test scenarios above (fig 2).

Fig 2

Also from the screenshot above, you can see that when I run these tests through Visual Studio Test Explorer (left hand side of the screenshot). I see some results related to these tests, which tests passed (green), failed if any (red), or not implemented (in yellow). Pretty nice, but as you might have thought, this list is gonna get really long as soon as you start piling up your tests. 

However, as you can see from the screenshots below, when you run the solution, you just need to provide a few inputs (Fig 3), or default to all (blank) and you can get a report similar to what is shown in Fig 4.

Fig 3

Fig 4

There are a few components to this - the Windows Form solution in Fig 3, the batch file that accepts inputs and of course MSTEST and SPECFLOW.EXE files. 

Configuring the solution

Once you lood the solution, you need to update the code behind the form, you can do this by right-clicking on the form once you load them in VS, or press F7. Scroll down to the lines that Load method in your form class and set up the paths to where you have your Test Project, the bin directory or directories where you have your DLLs, and where you want to store your TestResults.html file.

        private void SpecFlowReportGeneratorTool_Load(object sender, EventArgs e)

        {

            btnViewReport.Enabled = false;

            textSpecFlowProject.Text = @"C:\BlogPosts\SampleSpecflowProject\SampleSpecflowProject\SampleSpecflowProject.csproj";

            textDLLFile.Text = @"C:\BlogPosts\SampleSpecflowProject\SampleSpecflowProject\bin\Debug\SampleSpecflowProject.dll";

            textResultsFile.Text = @"C:\BlogPosts\SpecflowReportGenerator\SpecFlowReportGeneratorTool\bin\Debug\TestResults.html";

            initialResultFilename = textResultsFile.Text;

       

        }

The .csproj test file itself is just used to pick up the test project name, so its I included a few lines in the solution that you can use to configure this with your Company name and override the test project naming:

        private void btnViewReport_Click(object sender, EventArgs e)

        {

            if (MultiProject)

            {

                ReplaceTextInFile(textResultsFile.Text, "YourCompany Tests Execution Report", "SampleSpecflowProject Test Execution Report");

                Process.Start(textResultsFile.Text);

            }

            else

            {

                ReplaceTextInFile(textResultsFile.Text, "YourCompany Tests Execution Report", "SampleSpecflowProject Test Execution Report");

                Process.Start(textResultsFile.Text);

            }

        }

The actual replacing is done through the code below:

       public void ReplaceTextInFile(string filePath, string replaceText, string findText)

        {

            try

            {

                System.IO.StreamReader objReader;

                objReader = new System.IO.StreamReader(filePath);

                string content = objReader.ReadToEnd();

                objReader.Close();

                content = Regex.Replace(content, findText, replaceText);

                StreamWriter writer = new StreamWriter(filePath);

                writer.Write(content);

                writer.Close();

            }

            catch

            {

            }

The code above is again pretty straightforward, you basically just open the file, store it to the content variable and just do a Regex.replace() to search for a specific test, which in this case is the "SampleSpecflowProject Tests Execution Report" with "YourCompany Tests Execution Report". After the update, save it again and overwrite the same file, so when you open the HTML file the banner will read with your company name instead of the project name.

Once the program has all the parameters setup, it calls the batch file through the code below, with the parameters that we have assembled from the input text boxes.

        void worker_RunWorker(object sender, DoWorkEventArgs e)

        {

            start = new ProcessStartInfo();

            start.FileName = @"C:\BlogPosts\SpecflowReportGenerator\SpecFlowReportGeneratorTool\bin\Debug\SpecflowReportGenerator.bat";

            start.Arguments = ProcArguments;

            start.UseShellExecute = false;

            start.RedirectStandardOutput = true;

            start.CreateNoWindow = true;

            

            using (Process process = Process.Start(start))

            {

                using (StreamReader reader = process.StandardOutput)

                {

                    string result = reader.ReadToEnd();

                    Result = result;

                }

            }

        }

Putting it all together

Once you have the solution properly configured, the last step is configuring the batch file which is a cinch. The details are below:

@echo off

if Exist TestResult.trx del TestResult.trx 

if Exist %3 del %3

IF %2==-notags (

@echo on

"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\mstest.exe" /testcontainer:"%4" /resultsfile:TestResult.trx 

goto specflow

)

IF %2==-withtags (

@echo on

"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\mstest.exe" /testcontainer:%5 /resultsfile:TestResult.trx /category:%4

goto specflow

)

IF %2==-withmultipleproj (

@echo on

"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\mstest.exe" %5 %6 %7 %8 /resultsfile:TestResult.trx /category:%4

"C:\SpecFlow\SpecFlow_v1.9.0_bin\tools\specflow.exe" mstestexecutionreport %1 /testResult:TestResult.trx /out:%3

echo Created results file - FixedResult.html

goto end

)

:specflow

"C:\SpecFlow\SpecFlow_v1.9.0_bin\tools\specflow.exe" mstestexecutionreport "%1" /testResult:TestResult.trx /out:"%3"

echo Created results file - FixedResult.html

:end

You need to set the path to where your Visual Studio is installed, specifically where MSTEST.EXE is installed. You also need to put in the correct path to where you copied/extracted/installed Specflow.exe. After that you're set.

The batch file is the one that does the heavy lifting of creating a TestResults.trx file which is normally generated when you run a test. It then goes to the appropriate option (-notags, -withtags, -withmultipleproj) so it can call MSTEST and SPECFLOW with the parameters you pass from your solution. Lastly, it generates a test results file (or FixedResult.html) that will show you the results of your test execution similar to what's shown on Fig 4. 

Conclusion 

Remember those tags i mentioned earlier? Those same tags can be used as inclusion parameters when you run your tests. Say for example, you want certain kinds of tests as part of Smoke Test, you just tag your Specflow scenario with SmokeTest (or any name that describes what the group is for) and input that name in the Inclusion Tags textbox when you run the solution, and voila! It will show you only those tests that fit that description as below:

Fig 5

And there you have it, a nice looking reporting tool using MSTEST/SPECFLOW with a few enhancements through our solution.

If you like this article and want to get your hands on the full source code for this plus the sample test project, just email me with your email address and I'll send it to you. 

Happy Coding!