Pages

Showing posts with label Testing. Show all posts
Showing posts with label Testing. Show all posts

Friday, March 21, 2014

My TDD Happiness Graph

I'm helping my friend Mayur with TDD and he has his frustrating moments as is evident from his tweet

This happened with me when I was learning TDD. This happens with everyone. Key is not getting demoralised and keep going. I did not know this in the beginning and I gave up. Then came back again. I wanted to share this experience in best way possible and I came up with follwing - My TDD Happiness Graph

Take a moment and look at the graph. There are few interesting things in there

  1. It takes time to master the art of TDD (I am no way close to mastering the art yet). 
  2. In the beginning you would have lot of "turn off" moments with TDD. More experienced you are, more "turn off" you would have
  3. The real kick start to my TDD learning was when I paired with someone who knew how to test drive my code
  4. For long time, I was only test driving my C# code, leaving my JavaScript to mercy of time. It took me a while to realise that JavaScript can be tested with same effectiveness as any code in statically typed language. The tooling support for testing JavaScript code is not spectacular but it is not disappointing as well
  5. I have indicated that my happiness took a dip with dependency injection (DI). That may be misleading and hence clarification. TDD makes it easy to inject dependencies and mock them during tests. This made life so easy that I made mistake of applying DI without giving consideration SOLID and code maintenance. Overtime I learned how to tame DI but it is worth mentioning mistakes I made
  6. In the end, I have shown as having reached "TDD heaven". I do not know what this means, it just something came to my mind. This is a state where every time you code or think about code, you start with tests. Things like SOLID automatically follow or at least TDD makes it easy to refactor your code (brutally if needed) in order to achieve SOLID. 
One last thing, this is my experience with TDD and your experience may be different. But I am sure you had your own happiness crests and troughs in the journey.

I was finding it difficult to make Visio and blogger work nicely with each other but I failed. If you are seeing the image blurred then a high resolution version is available here

TDD Happiness Graph


Friday, March 7, 2014

Is manual testing dead?

Personal disclaimer - I work in a team where every developer follows TDD and is capable of writing browser automation tests in BDD style using Cucumber/Ruby. We have a manual tester on team and no automation tester. Other teams around me that I closely interact with have not manual testers on team but only automation testers.

TL;DR; - The world of programming is very fast moving towards automating everything under the sun. But IMHO we are far from calling manual testing dead. 

The objective of the article is not to reason about/against the above argument. Automation is a big investment and is good investment on most counts but it may not pay off all the time. It can be useful understand in what situations that investment does not pay off. There is no thumb rule to these situations, these are purely based on my experience working with software I have built over the years. But before we get into that, lets try to understand what does it cost to automate a test (or use case, user journey, scenario, whatever you name it). Formally this is called  "Total Cost of Ownership" of automating a test. This consists of following

  1. Cost of automating a test
  2. Maintaining the automated test over time as your code changes
  3. Cost of manually testing edge cases not covered by the automated test

This inherently indicates that manual testing is still not our of order, but lets talk on that later. The first point is interesting - cost of automating a test. If you are automating simple tests through unit tests or integration tests, then this cost is very minimal. The investment here pays off very well and you can afford to automate a large number of tests. The reduced cost is a direct outcome of maturity of modern IDEs and testing tools. Today, you can hit the ground running with a unit or integration test in no time. It does not take long time to build and run tests. It is becoming more and more easier to repeatedly run these tests on more than one machines (thus making continuous integration cheaper). These tests are almost entirely written in the same language that the software under test (SUT) is written in. That makes it even easier. Maintaining these tests as SUT changes is not hard. 

Things become little difficult when we move towards browser automation tests that let you automate complete user journey. Agreed, this field is no more new and lot of advancement has happened in last few years in this area. We now have very stable tools like Waitr and Selenium WebDriver that make programatically interacting with browsers a breeze. But there are still some challenges 
  1. A different set of skills are required to build these tests. If your team is new to this kind of testing then they would take longer time in the beginning to automate simple user journeys
  2. If the SUT changes, then it takes longer to make changes to these tests
  3. In my experience, people prefer using Ruby for automation. So if you are a .NET or Java developer then you need to invest in cost of learning a new language
  4. These tests are not as reliable as unit tests are. They may be flaky and lot of times their reliability depends on factors beyond just tests e.g. load on the machine, response of the web pages etc.

By now, you would have some clarity on what I mean by TCO. A test might be cheap to automate in the first place but maintaining it over time may be expensive.

Another important point of view when it comes to automated tests is how is the faring in detecting defects or how badly the test is hiding defects. As your code changes, your tests either do not change, change or become completely irrelevant. If your tests are not changing then it is possible that they are failing to detect defects or hiding defects. If a test becomes completely irrelevant after a code change then the investment you made in automating the test ceases. From this standpoint, it would be worthwhile to ask yourself, how frequently the code being tested is going to change. You might need to talk to BAs and product owners to get a clear answer to this but if you are confident that code is likely to change a lot in near future then you need to be judgmental about how far you go with automation (unit tests are exempt from this. I always write unit tests no matter what)

Having said all of the above, I feel, there are situations when we come to a conclusion that manual testing is a better option. Below is a list of such situations from my experience

  1. Testing long running batch jobs - It is difficult to completely automate the testing of long running batch jobs. The core logic of the batch job can be unit tested but automating a single end-to-end run of a batch job, though possible, is not feasible and not worth the investment. Even if attempted, in my experience, it does not give enough confidence about the quality of the software and we resort to some level of manual testing in the end
  2. Testing email content and delivery - Email delivery can be tested to some extent by writing emails to disk instead of delivering them but then you are not really testing delivery. Also testing content of the email and how they render in browser based email clients vs. desktop based emails clients is something that cannot be automated reliably enough. Moreover, there is no one standard when it comes to desktop and mobile phone clients. Combinations are just too many and investment in automating this with tools available today is not worth it.
  3. Testing of user interfaces - Browser automation tools confirm the present of a particular element on page but they cannot verify the look and feel of the element. There is some movement happening in this area and people are experimenting with different tools. But again, given the maturity of these tools, automating such testing is not reliable. It is best left for human eyes. Here is a list of tools that let you automate testing of visual aspects of a web page by the way.
  4. Browser compatibility testing - Gone are the days when everyone on the planet used IE6. People now use tens of different types of browsers and every product owner wants his product to work on any browser out there in the market. They do not want to loose revenue because we did not support a browser that a prospective client was using. So testing software on vast range of browsers is inevitable. If you have got some browser automation tests then you can go a step further an run you test in multiple browsers. There are tools like BrowserStack that let you do exactly that. But this approach is not scalable given the time it would take to run all your tests on all possible browser combinations. And usually, the number of user journeys automated are not large enough to build confidence that software works in all browsers. 
  5. Testing responsiveness of website - "Responsive" is the buzzword today. Every website you build has to be responsive (well, not every site, but most sites).  Testing responsiveness of a website is very difficult to automates for the fact that you would need to render the site on different devices in order to see how your pages are scaling. You can use websites that render your site in different device resolutions to show you how your pages scales but automating that is not very easy. This kind of activity is mostly preferred to be done manually on real devices.
  6. Testing user journeys spanning multiple systems - Enterprise software usually have more than one component intricately communicating with each other. In order to reliably tests such systems, data flowing through one component into another needs to be verified. It is possible to break the problem down into logical units and test each unit with support of unit tests. But again, you feel the need of one test that runs an end-to-end user journey and ensures that everything is where it belongs. Automating such tests would be time consuming. Such tests would also take a long time to run.
Each of the above situations is something I have experienced. For some, I have attempted automating tests, frustrated and given up. For others, I had to rely on manually testing after getting issues from production users. All this makes me feel that manual testing is not dead. Rather role of manual testing has become more important. Automation has made it difficult for simple defects to creep in by mistake. What is there in the software are the defects that are not easy to find. So manual testers of these days have a huge responsibility of finding defects that automated tests have hidden. 

Sunday, October 27, 2013

Fluent controller builder updates

Yesterday I blogged about a fluent controller builder that I put together. You can read about it here. While building further tests using this controller builder I soon got into trouble. My initial design catered well for simple controllers which always have default constructors. So following code worked

var documentController = Build.Controller<DocumentController>()
                              .Having(b =>
                                {
                                    b.PostRequest().At("http://localhost/api/content");
                                    b.UsingDefaultRoute().HavingRouteData("controller", "document");
                                });

 A problem with this code is that it is creating an instance of the controller for me. This works fine when controller has a default constructor. As complexity of my application grew, I soon needed to inject various dependencies into my controller and above code stopped working. To fix the problem, I have introduced a new class that lets you configure a controller instance for use in unit testing. So you control the instantiation part and pass the instance of a configuration class that would do rest of the magic. Here is how the new code looks

documentRepository = Substitute.For<IRepository<Document>>();
idGenerator = Substitute.For<IGenerateId>();
urlBuilder = Substitute.For<IBuildUrl>();

documentController = new DocumentController(documentRepository, idGenerator, urlBuilder);

Configure.Controller(documentController).AsHaving(b =>
             {
                    b.PostRequest().At("http://localhost/api/content");
                    b.UsingDefaultRoute().HavingRouteData("controller", "document");
             });

Here my controller has three dependencies for which I want to inject stubs instead of real objects. This is easy because now I can create my own instance of controller and send it in to new Configure class that would prepare my controller for unit testing.

The nuget package has been updated to include this new change. You can find the details here

Saturday, October 26, 2013

Fluent controller builder for unit testing Web API controllers

This post is not about unit testing Web API controllers. A gentleman named Peter Provost has already written about this in enough details here. I actually followed his post to fix some issues with my unit tests. In order to unit test Web API controllers you need to set up a few things before you can start calling action methods.


  1. Populate Request property of controller with a new instance of HttpRequestMessageClass
  2. Define a route
  3. Create a route data collection
  4. Create an instance of HttpControllerContext using the above and use this to set ControllerContext property on our controller instance
  5. Tell the request object created in first step to use the above configuration.    
Does this sound complex? It does to me, so let me show you the code that actually does the above and you would know that this only sounds complex
private static void SetupControllerForTests(ApiController controller)
{
    var config = new HttpConfiguration();
    var request = new HttpRequestMessage(HttpMethod.Post, "http://localhost/api/products");
    var route = config.Routes.MapHttpRoute("DefaultApi", "api/{controller}/{id}");
    var routeData = new HttpRouteData(route, new HttpRouteValueDictionary { { "controller", "products" } });

    controller.ControllerContext = new HttpControllerContext(config, routeData, request);
    controller.Request = request;
    controller.Request.Properties[HttpPropertyKeys.HttpConfigurationKey] = config;
}

This code is not very complex to understand but it has lot of moving parts. For example, I may not always test for a post request or I may not always use the same route. These things change based on the context of you test. So reusing this code in multiple tests is challenging. An easy way to fix the problem is to accept most things as parameters to the method. 

Welcome to the fluent world

Being a big follower of fluent way of writing code (read more about this here) I re-factored this code. Here is how the code for controller set up looks in fluent world
var documentController = Build.Controller<DocumentController>()
                              .Having(b =>
                                {
                                    b.PostRequest().At("http://localhost/api/content");
                                    b.UsingDefaultRoute().HavingRouteData("controller", "document");
                                });
Doesn't this read natural - "Build a controller named DocumentController having a port request at "http://localhost/api/content" and using default route and route data 'controller' having a value of 'document'". It indeed reads better and is easy to repeat in multiple tests with changing request types, routes and route data. 

But what is going on here. 

Lets analyse this code line by line. On first line we call  


Build.Controller<DocumentController>()
Build is a static class having a method Controller<T>. This gives us an instance of a class called ControllerBuilder<T>. Here is the code for both of these classes
public static class Build
    {
        public static ControllerBuilder<T> Controller<T>() where T : ApiController, new()
        {
            return new ControllerBuilder<T>();
        }
    }

public class ControllerBuilder<T> where T: ApiController, new()
    {
        private RequestBuilder requestBuilder;
        private RouteBuilder routeBuilder;

        public RequestBuilder PostRequest()
        {
            requestBuilder = new RequestBuilder(HttpMethod.Post);
            return requestBuilder;
        }

        public RouteBuilder UsingRoute(string name, string value)
        {
            routeBuilder = new RouteBuilder(name, value);
            return routeBuilder;
        }

        public RouteBuilder UsingDefaultRoute()
        {
            return UsingRoute("DefaultApi", "api/{controller}/{id}");
        }

        public T Having(Action<ControllerBuilder<T>> builder)
        {
            builder(this);
            return Build();
        }

        private T Build()
        {
            var configuration = new HttpConfiguration();
            var routeData = routeBuilder.BuildFor(configuration);
            var request = requestBuilder.Build();

            var controller = new T
            {
                ControllerContext = new HttpControllerContext(configuration, routeData, request),
                Request = request
            };
            controller.Request.Properties[HttpPropertyKeys.HttpConfigurationKey] = configuration;

            return controller;
        }
    }
There is a lot going on here..!!
 Actually not, it is same piece of code that we saw in the beginning in private method Build of ControllerBuilder class. All other methods are there in order to provide a fluent way of configuring and setting various pieces of information required to build the controller. There are two other classes referred here, namely RequestBuilder and RouteBuilder. They too are simple classes with methods that let you configure a HttpRequestMessage and route definitions in a fluent way. Here is how those classes look
public class RequestBuilder
    {
        private readonly HttpMethod httpMethod;
        private string requestUrl;

        public RequestBuilder(HttpMethod httpMethod)
        {
            this.httpMethod = httpMethod;
        }

        public RequestBuilder At(string url)
        {
            requestUrl = url;
            return this;
        }

        public HttpRequestMessage Build()
        {
            return new HttpRequestMessage(httpMethod, requestUrl);
        }
    }

public class RouteBuilder
    {
        private readonly string routeName;
        private readonly string routeValue;
        private readonly HttpRouteValueDictionary routeDataCollection = new HttpRouteValueDictionary();

        public RouteBuilder(string routeName, string routeValue)
        {
            this.routeName = routeName;
            this.routeValue = routeValue;
        }

        public RouteBuilder HavingRouteData(string name, string value)
        {
            routeDataCollection.Add(name, value);
            return this;
        }

        public IHttpRouteData BuildFor(HttpConfiguration config)
        {
            var route = config.Routes.MapHttpRoute(routeName, routeValue);
            return new HttpRouteData(route, routeDataCollection);
        }
    }
As you can see there is nothing complex in these classes, just a few methods that accept some parameters required to configure the object they are building and a method that actually builds the object.

In closing

Fluent interfaces, in my opinion, come in handy where an operation/task consists of series of steps that when written in traditional style feel unrelated. This is mostly found in the set up phase of actual operation. Hope you find this little tip helpful.

These files are also available for download as a nuget package named "FluentTestingUtils". If you are new to nuget then navigate to pacckage page on nuget to know more

Update

I just published an update to this package as I found out a tricky situation with using these classes. You can read the details here. I hope to push more and more updates in future.