go back to the blog

Dot responds to Test Leaders Live attendee questions

  • 03/10/2012
  • no comments
  • Posted by EuroSTAR

Q (i): Miriam, it could be interesting to ask Dorothy if sharing her ideas with Gojko would lead to a kind of new view or approach on test automation. I understand her opinion about context which is relevant in the way the tester AND developer acts. But I would wish that there won’t be all kind of opinions about automation in the world; maybe a more generic approach can be defined?

Q (ii): I recently attended a meeting with Gojko Adzic about Specification by Example. He also talks about test automation and is convinced that automation should be done by developers. So that testers can continue their very good manual work. Do you agree with that, and did you already cooperate with Gojko about this subject and his SBE-approach?

(from Gerrit Jan ter Harmsel)

A: Hi Gerrit – thanks for your questions. Yes I know Gojko, like his ideas and have both of his books. I like what he say, for examples about having just one form of documentation for both manual and automated tests – this is what I recommend also (p 31 of Specification by Example book). There is lots of very useful advice in Chapter 9 also. I haven’t read the book in detail as yet, but at first sight it certainly looks like we are in great agreement.

As I mentioned, it is important to be aware of context when talking about automation – things that are ideal in one context may not be workable in another. Perhaps this is why some people are adamant that all testers must know how to write code (if they assume agile teams with multi-disciplinary people) but those in for example a large financial institution would lose the best testers from a business and custoemr point of view if they required all testers to write code.

I would be quite happy with Gojko’s approach of having developers being the automators, leaving the testers to get on with manual testing, as long as the testers could also write and run automated tests as needed.

Q: Does your book contain example of sucessful using of DSTL by manual testers and that DSTL is convenient language which doesn’t require much efforts to learn it?

(from Evheniya Zhuchenko)

A: There is one anecdote that specifically mentions using a DSTL (Ch 29, p 552), but unfortunately, circumstances prevented them from getting the value from this that they had hoped. The levels of abstraction I talked about are discussed in chapters 2, 11, 12, 14, 15, 20, 21, 27, as well as 29.

Someone who has done good work in this area is Martin Gijsen ( – he has a good paper on keywords and DSTL.

Q: Do you have any suggestions on book on frameworks?
Q: What are your thoughts on Axe automation tool?

(from Richard Forjoe)

A: I’m not aware of any books on frameworks, but would be very interested to hear about any!

Axe is one of the commercial framework tools, which is included in a list of current framework and execution tools I recently put together (copy emailed to you on request!) I don’t know any detail about Axe (or any other tools) but have been aware of it for a few years.”

Q: How can exploratory testing be automated? Aren’t those antinomic?

(from Philippe Antras)

A: When I first heard about this, I though it was an oxymoron also, but it is actually an exciting new way of leveraging automation on a large scale. This is a type of testing where the inputs are generated from a random or pseudo-random point (which is what monkey testing also does). But ETA also requires some kind of test oracle (the source of what the correct answer should be) – not just “is the system still running” which is what random/monkey testing is looking for, but is the answer returned somewhat sensible. It’s not an exact oracle, but can eliminate things that are probably ok from those that are definitely wrong.

Harry Robinson, Doug Hoffman and Cem Kaner are all people working on this type of testing, where you can test a lot more of a huge complex system than you can with “hand-crafted” tests. This type of testing is automated (including the oracle) and it is exploratory in the sense that the next test is not planned in advance so is going to places not thought of.

Q: How do You convince the project management to invest in test automation at the same time explaining them it will NOT cut budget, it will “ONLY” improve the quality? That’s a typical problem I’m experiencing.

(from Michal Tomczewski)A: Actually, automation will NOT improve quality! Automation is a way of running tests. But even testing will NOT improve quality! Testing is a way of assessing quality and giving information. What improves quality is building fewer defects in in the first place, and taking note of things found in testing to not make the same mistakes in the future (so testing can indirectly improve quality if improvements result from bugs found).

This is what I was trying to say in my first point, that automation does not find bugs. I guess I didn’t explain it very well! If this explanation didn’t help either, please download an article from my website about objectives for automation – I hope that will help!

Q. I find it hard to convince Top Level Managers importance of testing, they think developers are the ones who are useful, and testers just an overhead. How should a tester act / work / talk so that top level managers are convinced ?

Q: Can we have example of ROI reports?

(from Binayak Silwal)

A: ROI spreadsheet sent to all those who requested it by email.

Probably the best way to illustrate the value of testing is by showing the cost of bugs that are missed by testing, and the number of bugs that testing finds (compared to those it misses). One of my favourite metrics is DDP – Defect Detection Percentage, which measures the bugs found by testing compared to those plus any found afterwards. This needs to be combined with a measurement of the cost to fix a bug when it is found by testing compared to when it is found afterwards.

Q: I find it hard to balance between becoming great manual tester and on the other hand a great automator. How do you feel about this? Is it possible for basically anyone to become an expert in manual testing and test automation?

(from Aleksis Tulonen)

A: I think it is great if you can be both a great manual tester and a great automator. If you have the skills in both areas (and they are different skills) and enjoy doing both, then I think you will be of great value for your company – go for it!Faisal Iqbal “Q: I have observed that with time companies are more into hiring programming skills testers then to have real good skilled tester. Do you think in future testers jobs will shift to developers ?

Q: I have observed that with time companies are more into hiring programming skills testers then to have real good skilled tester. Do you think in future testers jobs will shift to developers ?

(from Faisal Iqbal)

A: Yes, there is definitely a trend for companies to want to hire testers who have programming skills. It was very interesting that 16% of the webinar audience thought that all testers should also be programmers! I don’t agree with this – I think there are many great testers out there who do not want to become programmers, might not be very good at it, and wouldn’t enjoy it. If you tell them that they aren’t “”proper testers”” any more, then you will lose a tremendous asset for your company!

So I see that this is a trend, but I think it is an unfortunate trend.

Q: Its a great book! just finished reading

(from Jakub Jarosz)

A: Thank you very much! ;-)

BTW, I would be interested to hear from the 2% of webinar audience who didn’t find the book helpful, and why – thanks!

Q: Second Poll is a bit difficult when you’re working on getting test efforts going from scratch and you don’t really fit the groups shown in the poll

(from Jane Larsen)

A: Fair comment – sorry about that! I guess you were “Other”?

Q: selling the testing idea still having a bit convincing problems, and automation is an appealing word for clients, however rather than the major ROI aspects e.g. test time, data / code coverage what else would a sensible and reasonable KPI to benefit the client?

(from Amr Marwan)

A: Not sure I understand the question – you are asking what the benefits of automation would be to endusers/customers?

The main benefit would be that the software you are delivering should be able to be delivered to more consistent quality levels in less time, so you are able to give your client a better service by making efficiency savings yourself.

Not sure if this answered what you asked – email me to continue the discussion if you want.

Q: That explains everything! Vista was U-G-L-Y! What are your experiences with how the results of automated tests are interpreted? I get nervous when I find few bugs, but automated tests will have diminishing returns. How do we communicate this to “management”?

(from Stefan Gedal)

A: The value of automated tests is not mainly in finding bugs, although any bugs found are of course useful to know about. But if you have automated regression tests, they don’t find many bugs anyway, so automating them doesn’t change that.

Getting nervous when you don’t find many bugs is a good thing to do with testing, especially if you main aim is to find as many as possible – but this would be less with automated regression tests, since the main aim is to confirm that things that worked before are still working.

One way to show the value of automation (not the only way but simple and useful) is to use “”EMTE”” – Equivalent Manual Test Effort. Whenever an automated test is run, you “”clock up”” the time that it would have taken had that test been done manually. You don’t have time to do the tests manually, of course, but the automation is covering some ground that could be done manually if you had the resource. There is a short article about this by Mark Fewster which I could send you – just email me.

Q: The message is’nt changed the last decade. Doe we reach the right people?

(from Eibert Dijkgraaf)

A: A rather cynical view, but I’m afraid that to some extent at least you are right. But one reason for this is that there are new people coming into testing all the time, and they don’t know the things that we have learned over the past years and decades. So there will always be some “”telling the same message””, a bit like children who are just starting school keep having to be taught to read. (Does this make sense?)

Q: we are following scrum ,in one sprint when stories are developd we manually test them – we don’t have enough time for automation in that sprint – what’s your suggestion regarding test automation in scrum?

Q: how can we show the benefit of automation- in which form?

(from Erfana Sikder)

A: For the answer to your 2nd question, please see the answer to Elbert’s question just before.

It can work well to automate in the following sprint – don’t leave it any later than that though. So sprint A you test manually and release. Then while Sprint B is developed and tested manually, you go back to the tests for A and decide which tests will become part of the regression pack and automate those tests (and test the automation). Then Sprint B is released and you start work on C. At this point you are manually testing C, automating tests for B and running automated regression tests for A. As time goes on, you manually test the current sprint, automate the previous sprint and benefit from automated regression tests for all the other sprints.

Having said this, Lisa Crispin recommends that the automation is always done within the sprint, and I agree this is better if you can. But sometimes, especially if you are new to agile development, the way I described can work!

Q: We automate the tests and add them to automated regression packs when it is verified by manual QA. Then those tests are run in the next sprints. is that the best way?

(from Amir Ghahrai)

A: See answer to previous question from Erfana about this, where I describe how this approach can work well. Whether or not is the “”best”” way depends on many things! While this may not be the ideal, it may actually be best for you.

Q: If automation runs tests, and tests find bugs, then by analogy, automation should find bugs :)

(from Amir Ghahrai)

A: Ok ok, but it is indirect! It’s not the fact that the test is automated that finds bugs, it’s the fact that it’s a good test (which happens to be automated so it runs fast). Now if you had said “by analogy, automated tests should find bugs”, then I might be more inclined to agree with you! ;-)

Q: What are good ways to gain more insight into what kind of tools and systems can be integrated and will actually work well together (apart from your book which is great!)

(from Rainer Deußen)

A: Well, I was going to mention the book ;-). (Experiences of Test Automation in case you don’t know)

There are many discussion forums on the internet – for example LinkedIn has a test automation group. Attend conferences (like EuroStar) to hear and meet people who have done something similar. Follow blogs about automation, become part of the EuroStar community!

Q: Where does your automation testing stop as essentially you’re building an application to execute tests including the set up and tear down of your environment? Would you write tests to test your test architecture, and tests to test those tests? Should there be testware architect testers?

(from Jamie Bowdidge)

A: Good question. Certainly automation does need to be tested – there are some very interesting stories in our book when that didn’t happy (“zombie tests”). But there will be diminishing returns after a few cycles of testing the testing of the automated tests.

I don’t think there can be any fixed advice about where to stop – just where it is sensible!

To view a recording of Dot’s webinar, entitled “Intelligent Mistakes in Test Automation”, visit theEuroSTAR YouTube Channel.

Blog post by

go back to the blog


Leave your blog link in the comments below.

EuroSTAR In Pictures

View image gallery