go back to the blog

Unreproducible – You’re wrong!

  • 20/12/2012
  • no comments
  • Posted by EuroSTAR

Let me leave you in no doubt, in case you were not convinced by the title. I don’t believe in unreproducible bugs. Not only that: having thought about it for a while, no tester that I know believes in unreproducible bugs. The concept of unreproducible bugs (defect/incident/failure or whatever term you are used to) is alien to that of the stored program computing. The whole of the software industry is built upon the idea that computer processes are deterministic. From an initial set of conditions, specific inputs or trigger events and a given ‘code-set’ (whatever that means), the answer will be the same no matter how many times the procedures are followed. The answer won’t change unless one of the three constituent parts changes.

I have worked with some very good software developers over the years. It is not only my experience about a nasty little ‘unreproducible’; developers have the same notions. If the cause of a problem can be found, then something can be put in place to prevent the unintended consequences. For that is what a bug is: unintended consequences, to something or somebody. Diagnose precisely what happened, and prevention (or ‘bug-fixing’) is possible.

The gold standard on bug reports is to give sufficient detail for the problem to be isolated, the cause found, and a correction applied without reference back to the originator. Detail should includes the steps undertaken to exhibit the problem (including as necessary, screen shots, data that was used, how you knew that it was wrong, etc.), and whether it is reproducible at will. These two small words are important. In writings about raising bug reports, Rex Black suggests that if you run the test three times, it is reproducible at will if the problem occurs on all three occasions, it is reproducible when occurring twice, and it is not reproducible if the occurrence is once or zero (the latter when the problem was found, and subsequently, three unsuccessful attempts made to see if it happened again).

If we say that a bug is not reproducible, in essence we are indicating that at the present time, insufficient detail is known about the whole set of circumstances. The bug is able to be reproduced, but we aren’t able to do it – yet. It is what Brett Gonzales called “conceptually reproducible” – if all the information were available, we could recreate the anomalous result in the test lab. However, at this moment in time, we don’t know enough.

Can we return to the second aspect of the gold standard for bug reports? This concerns the testability of a fix. There should be enough detail in the bug report so that the any of the test team can retest the amended overall solution (be it a change to one or more of the three parts; initial conditions, input / trigger events or the code-set), again without reference back to the originator. Have a look at bugs that you have raised. Do they pass this test? Can John or Jane Tester pick up your fixed bug, and be able to tell whether the fix applied has really resolved the problem? This even should be true if the test was ‘wrong’ – the expected result from the test was incorrect.

It always amazes me that testers are different to many other professions. If you get a group of doctors together, they do not talk about patients that they have killed. Probably never. That is the elephant in the room that no-one mentions. Get testers together, and discussions can quickly turn to horrible bugs, that almost (or in some cases, actually did) bring the system to a complete halt. Many times, it was a nasty little ‘unreproducible’, and how, and who, found out the real cause.

The software industry in general, and testing in particular, relies on the predictive nature of the systems that we are involved in. Expected results are vital for testers, whether these are formally created, or in the mindset of the tester at test execution time. If you believe in unreproducible bugs, you don’t give expected results their central place in the armoury of the testing professional. Unseasonal though the thought is, an unreproducible bug has the same validity as Father Christmas. Neither exists! (But don’t tell your children).

Blog post by

go back to the blog


Leave your blog link in the comments below.

EuroSTAR In Pictures

View image gallery