go back to the blog

21 Great Myths of Popular Testlore

  • 28/04/2011
  • no comments
  • Posted by EuroSTAR

Software testing has its popular testlore version, based on instinct rather than on measurements and experiment results. Some of the myth of this popular “pop-testing” are either not true, or true only under limited conditions. However, they are often believed to be true beyond doubt. Such myths are harmful, as they prevent people from thinking correctly, and achieving correct results efficiently.

The Testlore.

1. More testing is almost always profitable.

The more testing is done, the better quality, right?
Sure, there might be some weak correlation – provided all other conditions are equal. But since all other conditions are never equal, the opposite is true. By testing more, and longer, you may find some extra bugs, but this attitude means that more bugs have been created before.

What is profitable, and leads to better quality, is not more, but less testing. What is profitable is better processes, especially better requirements engineering, and bug prevention. Testing is just a desperate, expensive, last-minute attempt to limit the damage.

2. Incident reporting requires good interpersonal skills.

If you believe it does, you have a desperately inadequate incident reporting process. If your incident reporting process is good and appropriately automated, it minimizes the impact of interpersonal issues. The process, and the tool, automatically forces even most socially incompetent people to provide all needed information in a socially acceptable way. It is the process, not its users, which must have good interpersonal skills.

3. Testers should be involved early in software projects.

Does the fire-brigade need to be involved early in architectural design, so that it can plan its unavoidable huge last-minute frantic fire-fighting activities?
No: some fire-fighting experts should be involved early to plan fire-prevention schemes, and design sprinkler systems. And to minimize the need for the actual fire-fighters to be involved at all.

4. Web-testing tools, tools for dynamic analysis, load testing tools and static analysis tools are all different tool types.

Sure… but only if black, yellow, red and white people are completely different kinds of mammals! All the four tools listed above are just slightly different sub-types of one big tool family, the test robots, that is tools for automatic test execution.

• Web-testing tools are robots specialized in running Web-applications.
• Dynamic analysis tools are robots with big eyes (or ears) and little or no hands.
• Load testing tools are robots with exceptionally big hands and big eyes.
• Static analysis tools are robots that do not test running code, but source code or other models.

5. An IT department within a company should have defined annual budget.

If it does, a 100% socialist economy is created, where everything seems to be for free, the demand is much larger than the supply, permanent scarcity is created, and everybody spends a lot of time queuing and fighting for the resources.
Instead, an IT department should finance itself by getting paid for its services by the business units using / buying projects, new IT products, enhancements etc. There would be a great number of benefits of such a solution, for example:

• The most profitable IT projects would be realized, instead of those coming from the departments with most political clout.
• Businesses would have a very strong motive to provide very high-quality requirements.
• IT would have a very strong internal motivation to be lean and efficient, and run projects fast.

6. Quality can only be achieved at the cost of productivity.

The game between quality and productivity is not a zero-sum game, where if one wins, the other loses.
The relationship is win-win instead: the most effective way to achieve high productivity is by enhancing quality: better requirements, less re-work, less debugging, less regression testing, less maintenance. Bar shouting, imposing a brutal police force and working plenty overtime (just joking!) – Quality is the best way to productivity.

7. Developers do not like quality because it hampers their productivity

Some developers may .believe it is so… or they do not like quality because it improves their productivity to such a degree that there is no more room for last-minute tinkering, for things being “95% ready” for months, for the cozy, endless debugging. Developer productivity improves a lot with good requirements, good design, coding guidelines, automatic and obligatory static code testing and extensive unit testing.

8. Test design techniques are for dynamic testing.

Only 50% true – you use test design techniques to design test cases for dynamic test execution. But test cases used for static testing do not grow on bushes either – they must be designed:

• Checklists, rules and objectives for document reviews are designed (well, they can be designed on the fly, a kind of happy-go-lucky “exploratory reviews,” but designed they must be).
• Thousands of rules used for static source-code analysis must be created (designed).
• Rules for static model-verification, like: are there any dead states? Any redundant states? Any unachievable states – they must be designed!

9. Scripted testing hampers testers’ creativity.

Scripted testing may give a pretext for being asleep and still pretending some testing is done. Otherwise, the opposite is true: scripted testing is a prerequisite for creativity, because it relieves you of the need to think – during test execution – of all those tiny, boring, obvious and necessary test cases, and gives you time to unleash your full exploratory powers!

10. An important step in choosing a test tool is finding one that suits our programming language.

This is definitely not true about high-level testing, where you should absolutely do everything possible to test far above any implementation details.
To some extent, this is true about unit and maybe integration testing. However, language-dependent tool aspects must be taken into account only as the last step.

The important test-choice criteria are what kind of testing is to be done. The necessary, last step is to check whether the tool of your choice can work with your programming language/environment/style.
So if someone looks for “Java test tools,”- beware! I wonder if they know what they need to test.

11. Agile methods are only applicable in small projects.

Agile approaches cannot be used for development in projects with thousands of developers – this is true. However, projects with thousands of developers do not work anyway. The requirements for such products must be divided into hundreds or thousands of increments. The thousands of developers must be divided into agile-size, manageable teams.

You may object that there is no ready-made “Big Scrum” method for synchronizing all these small agile projects into one big product? Right – not yet, anyway. Well, I am afraid that in this case you need to become your own master, instead of Scrum Master, and think for yourself! Sorry…

12. Agile methods require Test Driven Development.

TDD can only be successfully used in a kind of agile-like project environment – this is true.
But you can have a 100% agile project (Kanban, Crystal, Scrum – whichever you like) without any degree of TDD.

13. It is impossible to produce test specifications before requirements specifications are ready.

In the good old waterfall days, that would be true. But we no longer live in that cozy waterfall world. In an iterative, incremental, spiral, agile world you may have to do just that – create test specifications before (some) requirements are complete.

It may be hard, and risky, but possible. For example, you may specify tests for one increment, before all requirements are ready; or design test cases for just a small unit; or take chances; or just treat test specification as your requirement spec, TDD-like.
Yes, it is risky, it may require re-work: yes, the world is a dangerous place.

14. Agile methods allow more flexible changes of the requirements during projects.

Sure, some prophets pretend that agile means a happy, requirement-free world.
In reality:

• Agile means requirement-driven, even if the requirements are gathered in a very sloppy, unstructured, end-user driven manner.

• The most popular Agile method, Scrum, actually mentions requirements all the time, even if they are for some reason hidden under a misleading code-name “backlog.”
• Agile development of the packets of the requirements is 100% feasible even in projects using heavy-weight requirements management practices and formal requirements modeling.

15. Domain knowledge is important for testers.

It may be helpful, sure. Having a happy family life is good for testers, too!
OK, some domain knowledge may be good for testers, perhaps to make their lives merrier? But requiring testers to have domain knowledge means that you hope to plug all deficiencies of your requirements engineering, and of the whole process, by the people at the end of the pipeline.
If you are unable to make your business people tell – through the requirements – what is to be built, you may try to plug the hole by having business-knowledge people at the end of the development process. Bad cars require very experienced test drivers!

16. Traditional development methods tend to produce excessive amount of documentation.

Documentation can be hidden under the carpet, but it will reappear (shall I say “raise its ugly head”?) sooner or later.
If you have very little or no requirements, design and code documentation? That means there will be a lot more documentation later on, in one form or another:

• A lot of badly-structured, oral / e-mail discussions during development.
• Plenty of bug reports.
• Overworked and over-stressed customer-support department.
• Piles of change requests and new versions.

Badly managed processes / projects have a lot of unnecessary documentation, sooner or later. Good projects have little documentation, regardless whether they are waterfall or mega-agile.

17. In the future, whole development process, including testing, will be fully automated.

In the future, whole development process, including testing, will be automated as much as sensible and possible, so that project participants will be able to spend 99% of their time on being creative, human and flexible – including designing better automation ways!

18. In the future, there will be much more freedom and flexibility in development process.

In the future, the processes will be like well-oiled and predictable assembly lines, so that there will be much more freedom and flexibility in designing the products we want these assembly lines to produce.

19. Testers should be inquisitive, curious and diligent.

Does that mean that other project stakeholders, for example system analysts or developers, should be obedient, dull, sloppy and lazy?
There are no special personality traits that can sensibly be assigned such varied professional group as testers. It is just a cozy myth, a kind of quasi-psychology.

20. Testers are not vacuum cleaners!

Testers are just that: they are vacuum cleaners, looking for the dust specs that other people missed. Therefore, some job redesign is needed here:

• It is very bad to rely too much on vacuum cleaning instead of not making a lot of rubbish in the first place.
• Therefore, we need less vacuum cleaners and more quality engineers – people who design the job so that as little vacuum cleaning as possible is needed.
• Because some vacuum-cleaning is always needed anyway, we should divide this necessary house-keeping burden more or less evenly between all, and avoid full-time vacuum cleaners.

21. It is best if someone else than the author tests the product.

There is a grain of truth in it. But no more than a grain.
In reality, the so-called independent testing is very necessary:

• In toxic organizational cultures with plenty of incentives to cheat.
• In bad organizations where project members are not so concerned about the final result quality.
• When there are financial rewards based on wrong factors, such as lines of code produced or the date product is shipped, instead of on quality and maintainability.
• When people are over-worked, stressed and thus extra prone to make mistakes.
• When there is too little test automation for “stupid” coding, requirements or otherwise typical mistakes.

The Author

Bogdan Bereza-Jarocinski, born 1957, was among SAST-founders (1995), among SJSI-founders (2003), SPIN-initiative (in Poland) 2006. Authored two books on software testing and engineering, many articles, some translations. Pioneered ISEB Foundation in Sweden (1999), then in Poland (2002-2004). Gave training for ISEB Practitioner, for ISTQB Advanced, for IEEE CS CSDP, for REQB Foundation, and for IREB. Contact: [email protected]


Blog post by

go back to the blog


Leave your blog link in the comments below.

EuroSTAR In Pictures

View image gallery