Track Talk W11

Test Intelligence: How to Find More Bugs in Less Time

Elmar Juergens

13:45-14:45 Wednesday 12th June

Software grows with every release. At the same time, release cycles get shorter and shorter. As testers, we thus face the challenge to test more and more functionality in less and less time.

Unfortunately, our historically grown test suites are often unhelpful, since they test too much and too little at the same time. Too much, since they contain redundant tests that incur execution and maintenance costs but provide little value over similar tests. And too little, because we lack tests for important functionality in changed or new code.

Instead of doing more and more work in less and less time, our tests are, thus, costly and slow while still letting too many bugs slip into production.

In order to solve this challenge, we need to stop treating our testing efforts like a black box. Instead, as our conference motto suggests, we need to better understand what we are doing here, and act accordingly.

In this talk, I will demonstrate how data from our own test and development process shed light into this black box and allow us to both optimize our tests and align them with development activities: test gap analysis reveals which code changes have not yet been tested by neither automated nor manual tests. Pareto optimization of test suites and Test impact analysis identify the tests that most quickly find remaining bugs. And history analyses of the version control system show where the most bugs occurred in the past, potentially revealing fundamental process issues that we need to fix.

For each analysis, I outline the research foundation and its limitations. I give examples and experiences from practical applications in different domains, and show how they helped teams to find more bugs in less time.