Bloggo back to the blog
How to avoid the testing Swiss Cheese Syndrome-->
As a lot of teams suffer all over the world from the “Testing Swiss Cheese Syndrome” so I believe it is time to share the information that we have collected. By the end of this post, you should be able to make a first diagnostic on your testing activities and reflect on adapted medication.
The famous Swiss cheese
For the ones that are not cheese specialist Swiss cheese is a generic name for cheese with holes like you can see on the following picture which is a slice of Swiss Emmental.
To introduce you to this syndrome, let’s think about all the testing activities going on during an application development. It is usually a subset of unit testing, integration testing, functional testing, automated testing, manual testing, exploratory testing, etc.
It is very unlikely that a single testing activity covers the whole application. That’s where I see the similarity with a slide of Swiss cheese: you can imagine holes in your test coverage, areas that haven’t been covered by one testing activity.
From development to deployment, testing activities are done by different people like developers for unit tests or QA testers for functional tests. They use different tools adapted to their own testing practices like unit tests frameworks (JUnit, …), web services testing solution (SoapUI, …), functional test automation tools (HP QC, Test Complete, Selenium, …), Test Management Solution (TestLink, HP QC,…).
All these testing layers are added on the top of each other, and each slice has some holes.
The Swiss cheese Syndrome
This syndrome appends when you don’t have an aggregated view of all these testing activities. In such a case, testing holes join their force to create tunnels where bugs and regressions can stay hidden until production.
Furthermore, it’s even more effective if you manage to map testing holes and code modifications because it is where new bugs or regressions will pop up.
This is related to the concept of Test Pyramid developed by Mike Cohn in his book “Succeeding with Agile”. Mike made one more comment explaining that “Testing is an investment and the investment we make at one layer should be influenced by how well testing has been done at the other layers”. However, this requires an aggregated perspective of the coverage of all testing activities to see what previous testing activities have well covered and what’s not yet been tested.
At Kalistick, we have developed a tool to help teams figure out if they are affected by this syndrome and treat it. I will present few key concepts.
First, you need to collect the code coverage of all testing activities. For instance Kalistick Testing Booster solution provides agents to collect coverage of all test activities on any platform (CI, functional testing, …). Second, you need to capture software changes in order to assess changes coverage. This could be done on source code or using the source control management (SCM). However, Kalistick provides an innovative application scanner that is as simple as an antivirus but does a deep application analysis to identify changes and impacts down to the instruction level.
Last, aggregating all these data provides a clear view of what have been tested and where are your testing holes. See the following dashboard as example; it distinguishes functional tests (manual & automated) and build tests (unit & integration test done within a continuous integration platform). The blue area represents parts of the application that are not covered by any test. The red area without any green coverage focuses on code changes that are not covered by any test.
According to how large are the holes, I suggest using a functional perspective to assess the related business risks and to identify which holes should be filled first.
Whenever you have prioritized holes to be filled you need to decide which kind of tests to add (Unit, Functional, Exploratory, Manual, Automated, etc.). For manual tests, Kalistick provides a Test Coverage ScoresTM that assess how much change each test will covers in order to focus on most relevant manual tests.
You can see some example on Kalistick platform (use demo/demo as login/password).
Swiss cheese large holes mean pronounced flavor so this is not a problem except that it doesn’t slice well and comes apart in mechanical slicers. However with software, large testing holes means bad smells and a high probability that the application falls apart when going live.
Please share your view on this testing Swiss cheese syndrome and any medication that you have found effective.
Marc Rambert co-founded Kalistick to move forward its passions for IT & innovation, and to spread all over the world the testing virus that infected him when he lived in Scandinavia and helped him to survive 20 years of software development.