Bloggo back to the blog
On Diversity, Representation and Standards in the Testing Business-->
I think I’ve attended nineteen EuroSTAR conferences over the years since the very first one in London in 1993. I have always tried to get to the show for the keynotes and track sessions of course but what draws me just as much is the terrific social networking (and I don’t mean Twitter). I have to say it is meeting really smart, interesting people from different and diverse backgrounds each year that always adds the most value for me. Talking late into the night (and morning) is a common experience at EuroSTAR shows. (This year, might be an exception for me, for obvious reasons).
Some examples that come to mind (and bring a smile to my face)…
In 1996, testing client/server was peaking and the Internet was only just coming on to the agenda. I was talking to some people about performance testing web servers. Two guys wandered over and joined in the conversation. The conversation went something like this…
“So you know all about client/server testing then?”
“I said yes, of course, I am a consultant you know”
“Well, we have a bit of a challenge… We have 125 systems written by 60 suppliers in 30 programming languages that communicate across a very high speed system bus… and the whole thing flies at Mach 2. It’s the Euro Fighter. If you are so clever, how would you test that?”
I thought for a moment, and suggested we retire to the bar…
Another time, I met people who tested mobility scooters. I wondered what on earth they were doing at a software testing conference. So I said, rather disinterestedly, “That must be interesting?”
“It is. Really. We have to wear crash helmets and body armour.”
“Why is that?” I had to ask.
“Well, the scooters are limited when they are sold to the public. We take the limiters off for testing. They can do 40 miles an hour. It’s quite dangerous sometimes, but great fun.”
At another show, I got talking to a group of people about test techniques and design and thinking and other stuff. We spoke a little about state-transition models and using them to drive testing. We sketched examples on the backs of napkins etc. Then one fellow, who had been quiet most of the time, chipped in.
“We test IBM CICS. Our state model has 25,000 states.”
I could go on. Every year I meet the most interesting people in the testing business at EuroSTAR. It is the diversity of the people and the systems that people test and the remarkable, innovative solutions that meet their challenges that interest me most. Is there a standard approach to testing? Of course there isn’t.
Now, the EuroSTAR conference has a proud history and there are many of us in this business for whom it is the highlight of the testing year. It is easy to think that we who attend represent, in some way, the testing industry and the testers in it. But, with all the diversity that we know exists, we really should not be so arrogant to think that.
I’m reminded of a STAR East conference I went to in Orlando in 2006. There was an after-hours ad-hoc panel session in one of the rooms. There was quite an interesting Q&A going on, but someone on the panel said in passing, “We represent the testing industry”. At this point, I had to voice a word of caution.
“Next door is the Orlando conference centre (a huge place that can hold 100,000 or so delegates in a single space). There is an SAP conference going on in there with 25,000 SAP professionals attending. Half of those guys spend most of their time testing, in one way or another. In terms of numbers, with just over 1000 testers at this show, and not many SAP folk here, I don’t think it’s a reasonable thing to say that we represent the industry.”
I cannot think of a single body or conference that represents the whole of the testing industry. The business domains we work in, the systems that are built, the test approaches and technologies we use and the people and cultures we work with day to day are just too different.
SAP is one, very large, example of an industry within an industry. There’s not a lot of interchange between SAP folk and, for example, the popular testing shows. The same can be said for the high integrity, airborne, defence and gaming industries and there are lots of specialist IT niches in banking, manufacturing, pharmaceuticals that have their own QA conferences – and so it goes.
Recently, I’ve been talking to some people in the Payments Systems business. It is an industry, with stringent regulatory standards (https://www.pcisecuritystandards.org) and an awful lot of testing involved, and given the volumes they deal with, much of it automated. It’s just one, significant industry, but largely unrepresented in the popular testing shows. It seems that every year, I discover a new industry that tests. A lot.
These industries or niches are all special in some way, I suppose and it’s a challenge to get them to attend EuroSTAR in numbers because they often have their own industry-specific shows.
Now, you are almost certainly aware of the challenge to the emerging ISO 29119 standard for software testing that is out there. An online petition ‘Stop 29119’ (http://www.ipetitions.com/petition/stop29119) is attracting quite a lot of attention. You can learn more about the standard here: http://softwaretestingstandard.org/. The objections span several areas. Summarised, the arguments are that there is disagreement and sustained opposition regarding the validity of the standard and there is no consensus towards it.
The ISST (http://www.commonsensetesting.org/) set up the petition. They adopt the Context-Driven Testing (http://context-driven-testing.com/) principles. They reject the notion of standards on principle. On the CDT web page, using a documentation standard as an example on the web page above, they state that the use of such a standard “…is not unusual, nor disreputable, but it is not context-driven”. And that’s fair enough.
The software engineering standards being used out there are a bit of a mishmash. Every industry that embeds software and/or is safety-related, seems to have its own. The use of some software engineering standards is almost enshrined in law for some industries (airborne software is one example). The corresponding testing standards aren’t a good fit.
One of the goals of ISO 29119 is to tidy up the situation and replace the current testing standards so the organisations who use existing standards will probably find the new standard useful.
An integrated standard (for want of a better phrase) for testing has been mooted for at least 15 years. I certainly remember public invitations to get involved in the new standard around the time the initiative kicked off in 2007. As recently as 2012, Stuart Reid, who led the standards effort, pitched a talk at EuroSTAR ( /conferences/session/342/iso-29119-the-new-international-software-testing-standard) explaining the standard and inviting people to get involved.
I was involved in the development (with Stuart, Dot Graham and others) of what became BS 7925 – a standard for Component Testing all of twenty-two years ago. Part I was a Glossary (to pull together testing terms into one place) and Part II was a definition of the standard test design techniques (both functional and structural) with an idealised process for component testing. I wasn’t that keen on the ‘generic process’ part as much as the glossary and techniques definitions. But in the 1990s, given that there was no standard terminology nor agreed definitions of the techniques – I thought it was a good thing to get involved in.
But a standard test process? I have never been comfortable with that. A standard test process focuses on exactly the same aspects of testing as the certification schemes do. They focus on Test Logistics. The core testing thought processes and skills, I believe are universal, but logistics vary from place to place. My paper on New Model Testing sets out my position on this here: (http://dev.sp.qa/download/newModel).
I don’t believe an ‘all-testing’ standard is feasible, although it might be possible to create one that works for an industry sector. Or at least the processes can be adapted for use in a company’s context. That to me, would be to use 29119 as a guideline rather than a standard, and that’s how most process-oriented standards are used. (You could say that’s context-driven. But I couldn’t possibly comment).
The ISO 29119 working group is huge, http://softwaretestingstandard.org/aboutWG26.php but of course is also tiny and not representative of the industry. But then so is the group of objectors, I suppose and they probably don’t work in the same industry sectors as the authors anyway. My wild guesstimate is there are perhaps up to one million and a half testers worldwide and that doesn’t cover the testing activity of developers which is much greater. There’s an awful lot more people out there than most realise – or admit and the hoo-hah is a small ripple in a big pond.
Certainly, the companies who will be looking to use the standard in some way won’t be listening to folk who are avowed opponents of standards. But also, ISTQB who have for some years been promising that when the standard emerges, they will adopt it for their qualifications are unlikely to listen too. So the certification scheme will be rooted, like the standard, in Test Logistics – which by my definition are different wherever you go. Hey ho.
So, my position is this. I don’t support the standard, but I don’t object to it either. It’s one small part of the rich, diverse, tapestry of our testing business.
Conference Chair 2014, Paul Gerrard
Paul Gerrard is a consultant, teacher, author, webmaster, developer, tester, conference speaker, rowing coach and a publisher. He has conducted consulting assignments in all aspects of software testing and quality assurance, specialising in test assurance. He has presented keynote talks and tutorials at testing conferences across Europe, the USA, Australia, South Africa and occasionally won awards for them.
Educated at the universities of Oxford and Imperial College London, in 2010, Paul won the EuroSTAR European Testing excellence Award. In 2012, with Susan Windsor, Paul recently co-authored “The Business Story Pocketbook”.
He is Principal of Gerrard Consulting Limited and is the host of the UK Test Management Forum and the UK Business Analysis Forum.