Combinatorial Testing

Combinatorial – Meaning:

What does the term combinatorial stand for? Well in the world of computer science combinatorial method refers to testing all possible combinations of a pair of input parameters. Such a technique facilitates a much exhaustive type of testing as the idea is to test a specific functionality or so by initiating an interaction among the input parameters.

Combinatorial is a term derived from mathematics. In mathematics there’s a branch of study called ‘Combinatorics’ which states counting structures of a certain type and size to find out whether a criteria is met and analysing the objects meeting the criteria. The idea is to figure out the total number of outcomes within a given sample space. The very essence of combinatorics is to determine selection, arrangement and operation within a finite system. That is, in simple terms combinatorics provides a certain number of ways to find a solution to a given problem.

In the context of software, however, behaviour of a software application could be affected by quite a few factors like input parameters, configurations, variables etc.

The idea is to combine parameters to check whether it leads to some fault or not because sometimes a fault may occur as a result of interaction between two or more components. When we talk about parameters, we mean a combination of program variables, functions, external files/applications, operating system, browser etc.

Pairwise Testing and its Effectiveness:

Combinatorial testing is also referred to as ‘Pairwise Testing’. Pairwise testing offers 100% coverage in the application’s code by combining set of input conditions. Combinatorial style of testing is based on an intuitive technique that enable testers to systematically consider all possible scenarios.

combinatorial testing image two

Combinatorial Approaches To Testing:

  • Category Partition Testing: Talking about testing, we can categorise inputs into valid and invalid ones. So the idea of category partitioning is to segregate test case value from the complete set of inputs for a certain condition/statement/path.

    The inputs include parameters (variables) and environmental factors (OS, database etc.). For each category we further need to decide its class of values – normal, boundary, special and error values

  • Pairwise Testing: Here we try to test a functionality by forming pair of inputs. Set of input conditions is simply a way to trigger some action on the basis of occurrence of an event. Combinations of input conditions are created to test whether the application generates the desired output or perform as per expectations.
  • Catalogue Based Testing: Testers apply their domain specific knowledge and assimilate their opinions and experience to decide upon test input values.

Why combinatorial Testing?

This branch of software testing facilitates us to figure out the various causes of failures. It is a kind of software failure analysis to uncover facts like –

  • Logical Errors: A logical error in a code is something that generates some abnormal or undesired output because the code written for a specific functionality deviates from the expected result.
  • Calculation Errors: Incorrect order of operators used to perform mathematical calculations (BODMAS rule applies to calculation in computer programming) or the calculations must adhere to some basic mathematical rules. For instance 2/0 gives a ‘divide-by-zero’ exception, if float values are operated upon and the return type is integer then it’s an erroneous calculation.
  • Inadequate Input Validations: Data validation is about maintaining conformance to its originality and accuracy. Hence when we talk about input validations we mean that an input must be strongly type, adhere to correct syntax, contains only permitted number of characters and so on.

Challenges in Combinatorial Testing:

  1. A vast range of combinations have to be dealt with.
  2. It demands investment of a lot of time.


In an endeavour to deal with some common types of problems prevalent in software testing, combinatorial testing came as a rescue. As we know an application involves a vast number of input conditions that could be tested but it leads to a lot of time and cost consumption. The solution to the problem was to figure out an alternative that would maximise the value of each test case while reducing the total volume of test cases. Hence combinatorial testing emphasises a smart combination of input values based on aforementioned approaches to testing. Some of the testing tools meant for pairwise testing are – IBM Focus, Hexawise, Jenny etc.