Bloggo back to the blog
Mike Bartley’s Community Choice Submission-->
Presentation Title: Innovations for Testing Parallel Software
Company: Test and Verification Solutions
Country: United Kingdom
Concurrent software exploits the availability of multiple CPU cores on today’s modern hardware platforms in an effort to improve performance. However, concurrency introduces new classes of bugs (e.g. deadlock, race conditions) and increases the number of different possible executions through the code. Research suggests that 10% of the bugs in concurrent software are due to the parallel nature of the execution. Unfortunately, our traditional testing techniques for sequential code will not find these bugs.
In this talk we discuss the industry drivers for moving to multicore hardware platforms and how this will impact how software is developed. The talk will cover in detail the challenges of testing concurrent software and discuss the innovations required in our software testing processes required to meet the challenges: in the types of tests we need; in the metrics we need to measure the effectiveness of our
testing; and the methods we need to debug test failures. The talk will offer possible solutions to these innovations. Such solutions and metrics are vital to the effort to ensure the quality of concurrent software.
The industry drivers for the move to concurrent software The unique challenges of testing concurrent software The innovations needed to ensure concurrent software is properly tested.
Every few years a new technology comes along that changes the way we develop software – for example the move from machine code to assembler to 3GL’s; relational databases; the internet and mobile applications. We are now entering the era of multicore.
We are used to seeing performance double every 2 years (known as Moore’s law after Gordon Moore of Intel) through faster clock speeds. Unfortunately power consumption is exponentially related to clock speed and so such speed improvements are no longer acceptable (battery life, electricity costs, green issues,…). So the solution now coming from hardware designers is to get performance improvements through multiple CPUs. Increased performance with linear power – problem solved!
Well – solved from the hardware perspective. The problem is now passed to the software community to make use of the extra CPUs. For example, your next server or mobile phone could have 8 cores (or CPUs) and your programmer needs to be able to write software that distributes the processing load over those cores. This is hard – very hard.
So how does all of this affect me the tester? The answer is VERY significantly. Parallel programming has been around for a number of years (mainly as a niche activity) and research suggests that it introduces new classes of bugs which add about 10% to the total bug count. So, you may say, what is the fuss? It is just a 10% increase in bugs – not particularly significant.
Well, the problem is that the new 10% are much harder to find. The new classes of bugs include non-determinism. This is where the precise order of program execution is no longer predictable (remember – it is now parallel rather than linear) and different executions can yield different results. Debugging such failures are notoriously difficult as the bug disappears when you try to re-create it – that is the nature of non-determinism! Other new bug types are deadlock, race conditions, livelock,…
Multicore will also necessitate a change in our completion metrics as the current ones do NOT take into account issues caused by parallel execution and the new classes of bugs mentioned above.
This paper is not for the feint-hearted. It discusses technically challenging issues related to changes in the way we need to test to react to changes in the way we program for multicore environments.
If you want more detail on this then you can access a lot of detail on this on the TVS company website at:
Dr Mike Bartley gained a PhD in Logic at Bristol University and an MSc in Software Engineering and MBA with the Open University. More recently he has added ISEB Practitioner Testing Qualifications in both Test Management and Test Analysis. Mike has been involved in software and hardware development for 20 years as both a tester and test manager with responsibility for the signoff of a number of complex products. Mike has a particular interest in multicore software and how this will affect the software testing industry. Mike regularly writes articles and delivers conference papers on testing and verification.