How do you decide which software tests to run and which not to run? If you ask this question to ten different companies, chances are that you will get very different answers. Probably you will find out that no one has really thought about it – ask yourself this question (and if you come to a different conclusion, I’m happy to hear from you!). Most of the time, some features are said to be important and the focus is on the fact that they are covered by the tests. However, the general rule is: the more tests, the better.
What happens in very complex systems – namely when all functions are important and prioritization is difficult? First, let’s have a look at a simple example application with few functions, but with interdependencies: a simple application for booking movie tickets. There are three different parameters: the movie, the seat type and the seat location. The inputs of these three parameters are on the final ticket and thus interconnected.
The number of test cases for the selection of three and four movies is calculated as follows:
For 3 movies: (#movie options)*(#seat types)*(#seat location)*(#button) = 3*2*3*1 = 18 combinations
For 4 movies: (#movie options)*(#seat types)*(#seat location)*(#button) = 4*2*3*1 = 24 combinations
This means that for a new film six new test cases are added – still quite manageable. When we add a new parameter, for example the additional choice of two bookable snacks, each with three possible inputs (popcorn, tacos, no snacks), the new number of test cases is calculated to:
For 3 movies: (#movie options)*(#seat types)*(#seat location)*(#snacks)*(#button) = 3*2*3*3*1 = 54 combinations
For 4 movies: (#movie options)*(#seat types)*(#seat location)*(#snacks)*(#button) = 4*2*3*3*1 = 72 combinations
The number of combinations for three to four movies now does not increase by six, but by 18. It is easy to imagine that the number of test combinations can become very large for today’s applications. This exponential increase of test combinations is called Combinatorial Explosion. Complete test coverage without optimization or prioritization is no longer possible.
In the following post, I present a method for optimizing test plans: Combinatorial Test Design.
Combinatorial Test Design – Overview
Combinatorial test design (CTD) is an approach combining several test design techniques for finding a small test plan that optimizes test coverage. The principle is based on a report from NIST which states that 65-97% of all bugs are triggered by the interaction of two parameters, referred to as interaction level 2. If a bug is found only through the interaction of three parameters with each other, this is referred to as interaction level 3, and so on.
Since most bugs are found in interaction level 2, you can optimize your test plan by only creating tests that check for the interaction of two parameters. I will show you how this works step by step using our example application.
Step 1 – Find Parameters
First of all, the parameters of the application have to be captured. For our application these are Film, Seat type and Seat location. Although this is very simple for our application, this is actually one of the biggest difficulties of CTD. Complex systems often have a large number of parameters, which are not always clearly identifiable.
Step 2 – Identify Values
Next, the identified parameters are assigned values. These describe which selection options exist. The following figure shows the values for our simple application.
Remember: If we consider all possible test cases, we have 18 (3*2*3) combinations to be tested.
For complex applications it makes sense to use equivalence classes and boundary values. This way the number of inputs can be further reduced. You can find more information about that here.
Step 3 – Create All-Pairs
After identifying the parameters with the input values, we can create the optimized test plan. For this purpose I use the free Pairwise Online Tool. Here I enter the parameters with their respective values and can generate and display the optimized test plan. An overview of existing tools can be found here.
Let’s have a closer look at our optimized test plan. Pairwise testing optimizes for interaction level 2, which means that only the interaction of two parameters are tested in each case. This means that each combination of two variables are only generated once. For example, the combination of Episode IV and VIP seats = true is only tested for one Seat location (in this case Front), not for the input value Middle or Back.
By using this approach we reduce our test plan from 18 combinations to a more efficient plan consisting nine test cases.
Step 4 – Profit
By following these steps and taking into account the usage of restrictions, equivalence classes and boundary values, an effective and efficient test plan is generated. This test plan provides a good basis for testers to decide which tests to subsequently run or automate.
Challenges and Additional Thoughts
Although CTD offers an optimized test plan for a given interaction level, there are some challenges to the approach in agile development environments. For example, the current tools (you can find an overview here) are often difficult to integrate into daily development schedules:
- DevOps trend: due to the constant evolution of software, parameters change very quickly and so do test plans. To create a new test plan with the help of CTD for every commit is not very efficient. However, CTD can help to create a first basic, optimized test plan, which serves as a basis.
- System evolvement: CTD is based on a mathematical optimization problem that is still being researched. Unfortunately, software systems evolve faster than combinatorial models due to the fact that adoption of research in industrial use cases take time. Therefore, combinatorial models cannot keep up with the requirements of today’s software, which makes it impractical sometimes.
- Lack of prioritization: Although CTD provides an optimized test plan, the question remains, which tests should be implemented first. Especially in very large and agile software projects, the complete test plan cannot be covered despite CTD. Prioritization still has to be done manually.
Despite these challenges, CTD provides a simple method for creation of effective and efficient test plans, which can be used by testers as a basis. It is particularly exciting that it uses a black-box approach and can be used in every development phase. This makes it highly suitable for test plan generation in early design phases and can serve as a basis for implementation using TDD.
Deciding on what tests to implement can be quite challenging. More complex software systems and the amount of testable combinations increase seemingly endless. One approach tackling this problem is Combinatorial Test Design, which makes use of the fact that most bugs occur by interaction of two parameters. By identifying parameters, their values and creation of all-pairs, an efficient test plan is generated. This test plan might not solve all problems a tester has, but it still offers a great basis for test creation.