Complete Custom Test Automation Framework - Using C#, MS EXCEL and Telerik Testing Framework - Part 1

In this series of posts, I will share with you a complete custom test automation framework that I built using Excel as a test driver, coded in C# on top of the Telerik Testing Framework, that you can use for your own test automation. You are free to extend this solution subject to GNU Public Licensing.

Okay, lets start....First off, when designing a test automation framework, you need to understand the AUT (or application under test) that you need to automate. Based on my personal experience (your's may vary depending on what your application is etc.), I have found that Unit or Acceptance tests are best when you have more access to your application's code and is a lot more stable and easier to maintain.

I am a huge fan of this diagram, and it's a great starting off point when discussing options available to you when you need to start writing your own automation framework (source: http://www.velocitypartners.net/blog/2014/01/28/agile-testing-the-agile-test-automation-pyramid/):

Fig 1- Agile Test Automation Pyramid

However, there are a lot of times too that you may need to write UI automation tests (integration tests) and understanding which tools out there suits your need, depends on several factors.

I've compiled a table below of some of my personal experiences with these tools:

Test Automation Tool	Pros	Cons
Coded UI	·        Tightly Coupled with Visual Studio. ·        Great for recognizing UI objects especially WPF controls and MS-native controls. ·        Can be integrated with other MS Testing tools – Load Test, Unit, Web Performance Tests. ·        You can code using either C# or Visual Basic	Tightly Coupled with Visual Studio. You can’t use it outside of Visual Studio. Needs some work in terms of recording and playback, especially when debugging code and managing multiple UI maps.
Test Complete	·        Superior object recognition capability. ·        You can code using a variety of scripting options – C#Script, Jscript, VBScript, etc. ·        Can be integrated with other SmartBear tools like AQTime (used for profiling and detecting memory leaks). ·        Has great UI for recording/playback and debugging code.	Not open source, no free option.
Telerik Test Studio	·        Superior object recognition capability. ·        You can code using a variety full blown C# code or Visual Basic code which is awesome if you are already coding in either languages ·        Has great UI for recording/playback and debugging code. ·        Works awesome when recognizing complex Telerik UI objects. ·        Has an open source testing framework – Telerik Testing Framework that can be used outside of the Visual Studio integration.	Has some minor nuances that you need to be aware of. A bit on the heavy side and comes with a lot of software/tools that you may not need or use.

I did not include in this table Selenium because although it is great, and would be my first go to tool if I'm doing test automation for a web-based app. It is not a full fledged tool (does not have a strong UI support for debugging your code), and obviously if you are learning to do automation, this can be difficult at first to learn as it forces you to code right away. QTP may be the grand-daddy of all UI testing tools out there, but I find a lot of issues with QTP when recognizing new UI controls. Plus it is extremely expensive compared to these other tools. So, unless you have very deep pockets to invest in you test automation right out the gate, this may not be the right tool for you.

After looking at the pros and cons of using these tools, I've decided to pick and use Telerik as my solution because of the following:

1. Tight integration with Visual Studio. And since the web-app that I was tasked to automate was built in .NET, this was a natural choice for me.

2. You have C# language behind you. You're not limited to just a scripting language, but the full stack of C# .NET technology behind you that you can use at your disposal. For example, you can use .NET reflection here with relative ease.

3. Has an open source option in the form of the Telerik Testing Framework. You are free to create and distribute your code (of course subject to some restrictions in the EULA) as you need.

In my old job, my primary responsibility as Senior Test Automation Engineer was to evaluate tools and recommend the best technology stack appropriate for our needs (and obviously also cost effective at the same time). Which is why we have a lot of unit tests, and SDETs code integration/acceptance tests using Specflow and MSTEST, and of course Telerik Test Studio as our UI testing solution.

In this post, we will start looking at how a similar (because the one that we're actually using back then for our company obviously cannot be made public here and of course uses a more advanced hybrid solution) custom test automation that I earlier built. 

Lets get started...

First you need to have Visual Studio 2012 or better installed in your machine. Start Visual Studio and create 2 Projects in Visual Studio. The first project is a class project, name it as CustomTestController. This project will contain all your code for running your test cases, we call this your Test Controller. 

Next, you need to create a second Visual Studio project, this time you need to create a Console Based Application Project. You should name this project as TestRunner, and this project will basically be your test runner, responsible for instantiating your Test Controller and managing your environment.

To understand further the design of this custom test automation framework, I created a diagram below with the different components working in this solution:

Fig. 2 - Custom Test Automation Framework Design

Other parts of the solution - Test Driver, Test Cases, Test Config and Object Repository are XML based files that are built and managed using EXCEL. In the next post, we will delve more into the structure of these files as well as how to manage them.

By the third post, I will post in Github the full source code for this solution, that you can fork and use as you need. Thanks and see you next time.

Pivotal Tracker API v5 - Create your own Visual Studio Plugin

If you're into Agile software development, then most likely you or your company is using a tool that will let you manage your user stories. There are a lot of Agile project management software out there and I've had great experiences with 2 of the most popular - JIRA and Pivotal Tracker.

In this blog, I will share with you a sample project that I created that will let you hook up to your project in Pivotal Tracker. I will go through the basics of connecting to the Pivotal Tracker API v5, getting your tokens and credentials and walking through the sample code I created which grabs all the user stories in my project and displaying them in Visual Studio.

Lets get started...

As I mentioned, Pivotal Tracker is a great agile project management software. A quick wiki on Pivotal Tracker:

"Pivotal Tracker is Pivotal Labs' software as a service product for agile project management and collaboration. In July 2011,^{[dated info]} Pivotal Tracker had over 250,000 registered users.^[7]

The tool includes file sharing and task management, velocity tracking and iteration planning; release markers; and progress charts. There is an API^[8] for extensions and third party tools.^[9]"

What I like most about Pivotal is the fact that you can extend its functionality and directly plug in using your development tool. Although I am showing you how to use Visual Studio to plug in to Pivotal, you can you whatever your favorite code language is.

To start, you need to have access to Pivotal Tracker to be able to go through this sample project. For this blog, I just created a Sample Project, with a 60-day trial version. This project may not work after 60-days because the trial may have already expired, but you can use the project with a few minor tweaks to access your own project.

Below, shows a sample Pivotal project, with a few user stories:

Our Visual Studio Plugin will grab all these user stories in Pivotal and show them in a Textbox. 

Get your user credentials...

First thing you need to do is get your credentials, specifically your API token. To do that you need to either use curl via the command below:

$ curl -X GET --user username:password "https://www.pivotaltracker.com/services/v5/me"

This would display a JSON result with a bunch of information. What you will be interested in is the entry for "api_token" and the "id" under "projects".

Another way to get this is by going through "Profile" under your login name on the upper right hand side of your screen, similar to below:

Once you have your token and projectid information, you can plug this into your code.

Creating your Visual Studio Plugin...

The first thing you need to do is you need to go through the API reference from Pivotal Tracker, this will provide a ton of information you need to get you started, as well as enhancing this solution to satisfy your teams/company's need.

You can access the v5 API from this location - https://www.pivotaltracker.com/help/api

The next thing you need to understand is that the Pivotal Tracker API v5 returns a JSON result. For example, to grab all the user stories from your project, you can use the curl command below:

$export TOKEN=99cbf9c263dcb63e5505260822d647bd

$export PROJECT_ID=1157418

$curl -X GET -H "X-TrackerToken: $TOKEN" "https://www.pivotaltracker.com/services/v5/projects/$PROJECT_ID/stories"

This will display all the user stories in your project in JSON format. Notice that the JSON result follows a common structure, displaying such elements as id, story_type, name, description, etc.

In your Visual Studio project, you need to mimic this structure so you can bring over the data from Pivotal back to your C# classes.

Next you can start creating you Visual Studio project as a Windows Form project and begin to create the elements of the project as from the screenshot below:

The form only has a single element, a textbox with vertical and horizontal bars enabled. Notice also that the project has C# classes for Label, Owner, and Story. This is what the structure of the JSON result returns when you send an API call.

Other classes may be necessary depending on what the JSON result comes back with from your request.

After creating your project, you now need to create a method that will send the request over via an API call to Pivotal, this is the main logic that takes care of getting this data, and its shown below:

 public string GetStories(string project)     {

            string url = "https://www.pivotaltracker.com/services/v5/projects/" + ProjectId + "/stories";

            var uri = new Uri(url);

            // Create a new HttpWebRequest object.

            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);

            request.Proxy = WebProxy.GetDefaultProxy();

            var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);

            httpWebRequest.ContentType = "application/json";

            httpWebRequest.Accept = "*/*";

            httpWebRequest.Method = "GET";

            httpWebRequest.Headers.Add("X-TrackerToken", Token);

            var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();

            

            Stream objStream;

           

            var streamReader = new StreamReader(httpResponse.GetResponseStream()).ReadToEnd();

            List<Story> stories = JsonConvert.DeserializeObject<List<Story>>(streamReader.ToString());

            string storylist=null;

            foreach (Story story in stories)

            {

                storylist += story.id + "," + story.name+Environment.NewLine;

            }

            return storylist;

}

Explaining the code ....

From the method above, the first few lines assembles the HttpWebRequest object that uses the URL that we will need to grab our stories.

The next few lines are important, they are responsible for the content, the header, and the type of request that would be sent over to the API:

            var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);

            httpWebRequest.ContentType = "application/json";

            httpWebRequest.Accept = "*/*";

            httpWebRequest.Method = "GET";

            httpWebRequest.Headers.Add("X-TrackerToken", Token);

Notice also that you are passing a Token variable to the Web Request header information, this corresponds to the token you got from the first curl command you executed or from the Token on your Profile inside Pivotal Tracker.

Next, you need to issue the command to send the request over, and store the result to a variable:

var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();

If you recall earlier, when you created the project, you had separate C# classes to handle the result of the API call, however, the data that's being returned is in JSON. How do you convert the data from JSON to proper C# classes? I used an open JSON deserializer framework from Newtonsoft as part of this solution.

Once that's plugged-in as a reference to your Visual Studio project, I used it to deserialize the result back as below:

            var streamReader = new StreamReader(httpResponse.GetResponseStream()).ReadToEnd();

            List<Story> stories = JsonConvert.DeserializeObject<List<Story>>(streamReader.ToString());

Notice that the result from the stream (in JSON format) is converted to string then deserialized as a C# class matching structure of a C# class you created:

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading.Tasks;

namespace ManagePivotalProject

{

    public class Story

    {

        public string kind { get; set; }

        public string id {get;set;}

        public string created_at {get;set;}

        public string updated_at {get;set;}

        public string accepted_at {get;set;}

        public int estimate {get;set;}

        public string story_type {get;set;}

        public string name {get;set;}

        public string description { get; set; }

        public string current_state { get; set; }

        public long requested_by_id { get; set; }

        public long project_id { get; set; }

        public string url { get; set; }

        public List<string> owner_ids { get; set; }

        public List<Label> labels { get; set; }

        public string owned_by_id { get; set; }

    }

}

Once you have this information, you can then iterate through the structure so you can return it back to a Textbox that you can display as a result:

            string storylist=null;

            foreach (Story story in stories)

            {

                storylist += story.id + "," + story.name+Environment.NewLine;

            }

You should see something similar to screenshot below:

There you go....you have now created a Visual Studio plugin that grabs all the user stories from your Pivotal Tracker project! You can do so many things with this solution, you can extend it further so you can update your stories from within Visual Studio back to Pivotal, which might be useful if you have SDETs or devs that need to update the user stories and send it back with the updates to your Product Owner. Another extension might be to automatically grab these user stories, definition, and acceptance criterias and create Specflow feature files so your SDETs and devs dont need to write the tests themselves in Visual Studio. There are a lot of possibilities that you can do with this solution.

Hopefully you enjoyed this post and if you need the source code for this project, just put in a comment here and I will email it to you. Have fun!