Bloggo back to the blog
Defect Detection Percentage: handle with care!-->
Defect Detection Percentage (DDP):
The number of defects found by a test phase, divided by the number found by that test phase and any other means afterwards.
(Source: ISTQB Glossary of terms used in software testing).
In software testing Defect Detection Percentage is one of the most commonly used metrics for the effectiveness of a phase in the test life cycle. As can be derived from the formal definition above it indicates what percentage of the defects present in a software product is detected by the test phase under consideration. Hence process improvement programs aiming at increasing the effectiveness of a testing phase often use DDP to measure the improvement achieved.
A simple example
A software development organization discovered that during the first three months after releasing a software product the users reported about 20 problems. As the test team found 180 problems this means that the DDP equals to 180 / (180 + 20) = 90%. The test process was improved by introducing some formal test design techniques. After the next similar product customers reported only 10 problems during the initial three months. The test team found 190 problems, so the DDP was increased to 190 / (190 + 10) = 95%. The higher effectiveness of the test team had led to rise of the DDP of 5%. The process improvement was considered successful.
But can the increased DDP always be attributed to the success of the improvement program or might other influences change the defect detection percentage in a way that it only looks as if effectiveness was increased?
Let’s consider a few hypothetical cases.
Assume a process consisting solely of a system test phase. System testing discovers 90% of the defects, i.e. DDP is 90%. This means that 10% of the defects resulting in a failure were discovered by the customer!
The test process was improved by introducing component testing and after a while the DDP for component testing amounted to 50% (which is fairly hypothetical, I must admit). With an assumed total of 100 defects there were 50 left to discover by system testing. System testing discovered another 40 defects yielding a DDP of 80%, i.e. 10% lower than before process improvement. The 10 defects that were released all were memory leaks that only caused failure after continuous operation during 3 months. They would never have been discovered by system testing nor by component testing, neither before nor after the process improvement.
If, in parallel to the introduction of a component test, the system test would have been extended with a life test, running the system continuously over a long period of time, revealing some memory leaks, maybe half of the memory leaks would have been discovered. Quite an increase in effectiveness of the system test. But this would completely have been masked by the defects found in component testing. The DDP of the system test now amounts to 45 / 50 * 100% which is exactly 90%, just as before the process improvement.
Still the process consists of only component testing and system testing. The DDP of component testing is 50%. The DDP of system testing is 80%. Given a total amount of 100 defects to be discovered this means that component testing find 50, and system testing finds 40, leaving the remaining 10 defects in the released product.
Now we improve component testing. It now finds another 5 problems (5 of the remaining 10 that would otherwise have been released to the public). DDP for component testing now is increased to 55%. DDP for system testing now is 40 / 45 = 89%. Looking solely at the system test level, we might conclude that its effectiveness been increase significantly. In fact the system test wasn’t changed at all: the same 40 test cases were executed, the same 40 defects were found.
Now comparing these three cases what did we see:
Case 1: DDP for system test decreased, while the system test effectiveness remained the same;
Case 2: DDP for system testing remained constant, while the effectiveness was increased immensely;
Case 3: DDP for system testing increased, while the effectiveness remained the same
Now that was easy. Can we also come up with an example that the effectiveness would decrease, but not the DDP?
Case 4: DDP for system testing remains constant, while the effectiveness clearly decreases.
Consider the following.
The component testing phase is improved. A strong focus on memory management has revealed the cause of all those memory leaks. During the component test the 50 problems mentioned earlier are detected as well. So DDP for component testing is now 60%. 40 defects of the initial 100 are to be discovered by system testing.
But system testing actually finds on 32 of them, which means the DDP is still 80%.
The eight defects released to the public would have been found by system testing, if the quality of the process had not deteriorated.
A bit more mathematically
Assume a process consisting of two test levels, Component Test and System Test.
After the system test the product is released; assume the phase after release is called Production.
Let’s define the following
N0 = the number of defects initially present in the product
NCT = the number of defects detected during the component test
NST = the number of defects detected during the system test
NP = the number of defects detected during production
NU = the number of defects not detected during any phase (i.e. never detected)
DDPCT = NCT / (NCT + NST + NP + NU)
DDPST = NST / (NST + NP + NU)
Let’s simplify this by assuming NU = 0, i.e. the system will be used a long time and eventually all failures will be noticed, so
N0 = NCT + NST + NP
DDPCT = NCT / (NCT + NST + NP) = NCT / N0
DDPST = NST / (NST + NP) = NST / (N0 – NCT)
The latter formula shows that an increase of DDPST can be caused by solely increasing NCT, i.e. by finding more defects during the Component Test.
Now this is interesting! One of the problems of testing during development, or even sooner by using Test Driven Development, is that not all, if any, defects found are registered.
Nevertheless the effectiveness of this early testing will show as an increase of the DDP of the next test level (all other things being equal).
Dropping the assumption that NU = 0 does not change the reasoning above, it only changes what can or cannot be measured. We cannot measure N0 since we will never know NU. On the other hand, is this important? When monitoring production sufficiently long, NU does not matter anymore. Problems that are not discovered during a long period of production are not relevant. These are those defects that never result in a failure.
Each test level has its specific purpose, be it finding defects in the code of tiny building blocks (component testing) or finding defects in communication between entire systems. The DDP measured for a specific test level can only be compared with the DDP of the same test level at some different time, but only if the level sticks to its specific purpose. When defects meant to be discovered at level A are discovered too late, say at level D, improving level A has an impact on the DDP of level D, although the effectiveness of level D is not altered at all.
Does this render measuring DDP useless? Maybe not, but it does imply that a careful analysis of the changes in DDP is absolutely necessary, especially in complex development models with multiple test levels. So be aware that an increase as well as a decrease in DDP of a certain test level may just indicate a change in effectiveness of some other test level.
Or would it be a good idea to simplify things and just look at:
– what problems did our customer find?
– should we have found them and could we have found them?
– and if so, how can we improve our testing to find them next time?
Piet de Roo has been engaged in software development and testing since 1996. He was employed as test analyst, test manager, test process manager and test automation manager mainly in the automotive branch focusing on embedded software in car radio and navigation systems. In various international roles he was responsible for software verification in traditional and agile processes. Currently Piet works at Improve Quality Services as a consultant, coach and trainer in testing, test management and test automation. He is an accredited trainer for the ISTQB Foundation and Advanced courses.
Improve Quality Services BV, based in the Netherlands, focuses on advanced and innovative services in the area of testing and quality management. It offers (international) consultancy and training services with respect to testing, quality management and requirements engineering. Improve Quality Services is an accredited course provider for ISTQB Software Testing Foundation and all ISTQB Advanced modules (Test Manager, Test Analyst en Technical Test Analyst), Tmap Next foundation and TMap Next Advanced and the “IREB Certified Professional for Requirements Engineering” course and exam. In the area of test process improvement, they are accredited by the TMMi Foundation to perform formal TMMi assessments.