Bloggo back to the blog
Inner Value of Test Cases – Wolfgang Platz-->
Test coverage – the measure of all things
Test managers, like every project manager, are required to justify their budgets by the outcomes achieved. The most important criterion for objectively determining test outcome is the test coverage achieved within the test. Test coverage is the quotient of the number of tested and available functions – by definition this is between 0 and 100 %. Meaningful key performance indicators (KPIs) can be derived from the test coverage (e.g. test coverage per se, test coverage/test case, costs/test coverage, test runtime/test coverage). These can then be used to measure the productivity of the test team. If the test coverage is unknown, then efforts to determine and control performance within the test are only meaningful to a very limited extent.
In determining test coverage, the first thing that has to be considered is that the significance of the different functions under test is scattered widely: according to the 80:20 rule, which also applies here, 80% of the business is handled by only 20% of the functionality. This fact can be taken into account by assigning risk weights to the individual functions. The risk weights are normed (= sum of the risk weights of all the functions = 1) and thus represent the overall contribution to test coverage that a test provides for the respective function.
Test coverage on the level of test cases
On the level of functional areas, the contribution of a function to the overall test coverage is equal to the business risk that occurs in the case of failure or errors in the function. If test cases are now created for testing the function, how do the individual test cases contribute to the test of the function?
Methods and tools have been available for a long time that can determine the code coverage of test cases. The code coverage expresses the percentage of the entire source code that is run through during the execution of a test case. The code coverage, however, is a technical measure and its meaningfulness from the business perspective is rather limited: the significance from the business perspective does not express anything about the amount of code required for the technical implementation of a function. And so far there have not been any reliable methods for determining the test coverage of individual test cases from the business perspective (= business-based test coverage).
The fact that so far the test coverage has not been able to be determined reliably from the business perspective has had immediate consequences for CIOs, project managers and IT managers: the common practice so far has been to justify the performance of the test team by the number of test cases executed. (Note: This is as if one would try to illustrate the distance between point A and point B by the length of the route one took on a trip to get from A to B.)
New approaches in test case design
New approaches in test case design provide exact rules for building test cases to maximize their test coverage from the business perspective. At the same time, these approaches ensure that the failure of test cases make it possible to draw clear inferences about the cause of error. The starting point of these methods is the established methodological concepts of the creation of equivalence classes and boundary values which are enhanced by different combinatorial concepts (orthogonal combinatorial generation and linear expansion). These concepts are a novelty and the key to increasing test efficiency.
The test coverage of individual test cases can be determined by first assigning weights to the business relevant degrees of freedom and their equivalence classes. Then the weights are normed appropriately and finally, aggregated, according to the actual values of the different degrees of freedom in the test cases, to the overall contribution of test cases to the test coverage.
Test coverage of a group of test cases
The new methods in test case design allow the overall contribution of each test case to the test coverage to be determined. However, what happens to the test coverage achieved when not an individual test case, but groups of test cases, which is usually the case, are executed for specific functions? Can the test coverage of the individual test cases simply be added up?
The astonishing discovery is that the test coverage of the individual test cases cannot simply be added up. The reason is that meaningful business-based test cases can almost never be built for the same functional area without redundancies.
Example 1: To be able to create an account, a customer must be assigned as the account holder, i.e. a new customer is entered or an existing customer is selected. If the function for creating a new account is to be tested comprehensively, various combinations in the sequence and test data are to be run through. An account holder is required for each new account created. Thus, running through the functional part of the customer selection results in redundancy.
While the first test case still makes a contribution to the test of the customer selection, testing the rest of the test cases is redundant in terms of this part of the function. The bottom line is that the overall test coverage of a set of test cases is (at least) less than the sum of the test coverages of the individual test cases.
When the test results are examined more closely, there are other surprising insights: If a test case fails, this does not mean that the percentage of error in the test result is necessarily equal to the test coverage of the test case. Considerations of redundancy also have to be taken into account here.
Example 2: In two test cases on creating an account, the customer selection is run through each time (see also Example 1). The first test case fails, but the second one runs through completely. This means that the customer selection can be run through completely – the percentage of error of the first test case to the overall result is thus less than its test coverage.
The fundamental innovation of being able to determine the exact value of individual test cases (= their business-based test coverage) thus does not go far enough to be able to derive directly the correct contribution of the test set to the overall test outcome.
A new method for determining test cases and test execution rules make it possible to control the redundancy issue in the determination of test coverage.
The prerequisite for this is the incorporation of happy paths (straight-throughs): happy paths are the paths through the application under test that have the lowest probability of causing an error. To reach a specific test objective, the respective test case is built in such a way that it exclusively consists of happy ends up to the objective to be tested.
The effects of this are that if a test cases fails, the path to the test objective has the lowest probability of being responsible for it failing. Thus, it is possible to say with the highest probability that the error lies in the test objective itself. All of the test cases exhibit similar redundancies: they are directed towards exactly one test objective and in addition they are built of happy path elements.
This rule for building test cases is referred to as linear expansion: A complete happy path is built for each function under test. This consists exclusively of happy path elements and is always run as the first test case in test execution. All of the other test cases are completely redundant with regard to the happy path up to the test objective. In this way, the contribution of the test case to both the overall test coverage as well as to the test result can be determined easily and represents the test coverage of the test case to the actual test objective.
Currently available state-of-the-art tools already offer a powerful tool for determining the inner value of test cases. Through the method described above and the timely and accurate assistance of tools one of the biggest deficits towards optimizing the efficiency of functional software testing has been eliminated.