Bloggo back to the blog
From the Webinar: The Evil Tester’s Guide to Technical Testing-->
Below are Alan’s responses to questions posed at the ‘Best of EuroSTAR Conference 2012’ webinar entitled: ‘The Evil Tester’s Guide to Technical Testing’ from Tuesday, 4th December 2012. View the webinar here in our webinar archive. You can also download a copy of Alan’s slides here.
Writing answers to these questions helps me revisit some of the concepts that I tried to explain in my EuroSTAR Webinar “The Evil Tester’s Guide to Technical Testing”. I hope they help expand on anything that might have remained unsaid on the Webinar.
I want to encourage everyone to look at how they do testing and ask themselves. Do I understand this application from a technical perspective? Am I checking my results at the most effective level of the technology stack (you can check results below the GUI)?
Some of the questions seem to stem from an classification of ‘technical testing’ as an ‘extra’ rather than as an ingrained part of the normal testing process.
Increase your technical understanding so you can identify more risks and make more informed decisions about risk mitigation strategies. continually learn more and you will identify more ways of testing, and more ways of ruling out certain tests.
Thank you for all the questions. I hope you recognise the value in expanding your existing technical skills, and use them appropriately as you test.
Q: Are we not going to repeat most of the Unit testing when you spend so much time on technical testing?
A: No I don’t think so. Unit Testing serves a very different purpose than any Technical Testing activities. Unit Testing checks the code at a low level, often with minimal integration between code components using mocks and stubs. Unit Testing runs as part of the CI to provide fast feedback on broken assumptions and unexpected impact of code changes.
If you do worry about this, then review your testing with the developers against the Unit Tests.
Q: Do you have any suggestion or art of pointing out that the code or SQL query is incorrect/without hurting Developer ego?
A: First – how do you know for sure that you have identified incorrect code or an incorrect Query? If you have evidence, then you share the evidence. If you have an instantiated example that specific input repeatably demonstrates an incorrect output, then share the example. If you have based your verdict on an interpretation of a requirement then you need to share your interpretation and agree which interpretation of the requirement the writer actually wanted. I would suggest you stop worrying about hurting someone’s ego and concentrate on putting together a message that either has no possibility way disagree with it, or if you identify scope for disagreement, then discuss your interpretation.
Q: Do you think studying programming languages is useful?
A: Yes. I study the programming languages used in the construction of the systems that I test. This allows me to read the code, contribute to the code, understand what type of defects the language’s coding construct makes easy and those it helps rule out. I can also use the language to construct adhoc automation to augment my manual testing and look at the application in more detail e.g. dive into the database, monitor message traffic. I consider useful anything that helps me understand the system I have to test in more detail.
Q: How do you justify doing more technical testing on projects ie additional costs of tools and its impact on timescales, how would you justify this to a project?
A: Most of the tools I use do not come with a cost: they tend to come from the open source community.
I don’t use the phrase “Technical Testing” to convey a ‘phase’ or an extra stage or additional testing. I use it to mean that all the testing I do, I underpin with increasing technical knowledge, I check results at a technical level as well as a requirement level. So it adds no additional impact on timescales because I test in this way so the time it takes for me to test incorporates the technical approach I take. If I find more defects as a result of testing with a technical view point then I don’t justify this to the project, I raise them as defects. I tell people how I test and the tools I use, I have never had anyone tell me that they don’t want me to look deeply at the system and the possible problems.
If you do encounter resistance to incorporating more technical skills into your approach to allow you test test more effectively then I think you should raise that as a project risk.
Q: How do you go about doing some technical testing if you don’t have direct access to the backend system of an application?
I approach all my work with as much technical skill as I can, at the levels that I do have access.
I would also raise lack of access to the backend as a project risk because we don’t have visibility into the full application stack.
Q: Do you think obtaining technical skills is dependant on the projects the tester’s work on? ie Technical testing limited to projects scope
A: I think the skills that you focus on in the short term should related to the project and technologies in use, because then you have the chance to put them into action and add value as you learn. But if the projects you work on don’t allow you to explore technologies that you do want to learn then you may have to use time outside of the project to learn them.
Add value. Follow your interests.
Q: Does technical testing rely purely on tools?
A: No. Technical testing relies on understanding and increased learning of the technical domain. This knowledge allows you to ask questions and identify risk, even without touching the application. Tools help us extend our reach into areas of the application that we can’t otherwise approach. e.g. you can’t see data in the database unless you use tools, but you won’t know why or how to do that unless you first increase your understanding about the architecture and learn something about the database in use.
Q: How do you overcome resistance from developers and system administrators who say ‘that’s not your job’?
A: I explain to them that my job involves me doing those things. I try and use the processes for gaining technical access. I pair with people who can do those things. I raise project risks about not having the ability to fully test the system. I educate people on what I can do and what I understand.
You might ask who does do the things you want to do, and try to check their results to review them for coverage. This might help expose a gap in the current test approach.
Q: how do you reach a compromise between efficiency and risk when technical testing?
A: I test from a technical perspective because it increases my efficiency as a tester. I can identify low level defects faster because I can see the impact at a technical level rather than waiting until the impact ‘might’ gain exposure through a GUI or follow on business requirement.
I test from a technical perspective because I want to target risk in the application.
I infuse my entire approach to testing with technical knowledge. If I don’t do this, then I have to raise additional project risks that the test approach has gaps in its aims and coverage.
Q: How do you start applying “technical testing” in an existent qa-department?
A: Start by surveying the ground and asking questions of your understanding.
Identify what you do, and how you can approach it with more technical knowledge. Ask questions about how much you understand the system under test and fill gaps in your understanding. Look at the tools in use, do they cover ever technology on the project? Can you observe, manipulate and interrogate at every level of the technology stack?
– Do you understand the architecture? If not then start learning.
– Have you read the code? If not, start reading it.
– Have you seen the configuration files for the application? Do you understand the configuration differences between environments? Do you know what risks those differences introduce?
I personally start with myself, my learning and my approach. When new on a project I try and fill any gaps in my knowledge by asking questions of the team. If they know the answers then we have those gaps covered, and we can see how effectively we cover it. If we don’t know the answer then we have a gap we then choose how to fill.
Q: Do you have internal trainings on how to apply technical testing or is it just about self-exploration?
A: When in organisations I maintain lessons learned and knowledge transfer documents on wikis. I encourage knowledge sharing sessions either with myself leading them or learning from other members of staff – preferably everyone showing what they know. Every project and system requires different combinations of tools, knowledge and approaches.
Q: How to you ensure you don’t miss out on issues that don’t require technical testing, possibly usability testing?
A: I consider coverage as something I have to manage independently of our approach, by stepping back and surveying what we have managed to do. Having said that, I think that even “Usability Testing” can benefit from increased technical knowledge about the application. What technologies aid usability, what don’t. e.g. How can I measure the speed and responsiveness of the system?
Q: In a model where tests are mapped to business scenarios, how do we ensure technical testing has traceability – thus ensuring we have done enough testing (testing doesn’t finish, it stops.)
A: I use whatever tools we have agreed on to write up my testing notes. I can add my notes to the business scenario test execution log. I can raise new risks. I can raise defects. I don’t do “Technical Testing” in addition to ‘other’ testing. When I test, I test with an eye to the technical architecture of the application so I test technically regardless of what I test or how I have to document it.
I hope that helps explain what I mean by Technical Testing more clearly, as I don’t see any conflict with increasing my technical perception while testing business scenarios.
Q: Is there a community of technical testers to learn from so that you could find out relevant stuff for your own project?
A: Most of the technical information I have does not come from ‘testing’ it comes from the wider world of Software. I survey the technologies in use, then go look at the official web sites, speak to the other team members (devs, architects, admins, not just testers). I have no doubt that there exist plenty of testers with massive amounts of technical knowledge, I don’t know that a separate community exists, or needs to exist, around them.
Q: More often than not, I experience projects that try to evaluate not when they have done FULL testing, but rather ENOUGH testing. Doesn’t technical testing risk being not done for economical reasons? Or is that merely a question about testing maturity?
A: Any project can increase the risk profile by restricting the skill sets of its staff for economic reasons e.g. stop unit testing because it increases the time to create code.
I don’t think I have ever done ‘full’ testing, as I understand it. I’ve never had time to exhaustively cover every data set and state combination or memory profile or etc.
I frequently have to evaluate if I have done ‘enough’ testing based on the time and budget available. Approaching the testing technically provides me with more information that I can use to evaluate ‘enough’ and I can assess risks differently. I might be able to use a technical understanding to re-evaluate the risk profile and reduce the amount of testing done in one area because I consider it ‘enough’ and re-prioritise in a different area. I consider it more of a risk not to understand the system from a technical perspective and approach my testing in that way.
If I viewed ‘technical testing’ as an extra, then I would have to prioritise it against the budget and time. I don’t view it as an extra, my normal test process involves technical understanding, use of tools, observing the system at different levels of the technology stack.
Q: Sometimes I get blinded by testing white box (or technically) and I don’t think rich test ideas. So, my approach now is to first test less technically in order to cover more test ideas. Though, when I get to technical testing, I often realize that some of my previous tests were useless. Alan, what’s your approach on this?
I multi task at the different levels of my models. So I might start by checking some basic functional flows. Then I might look at the database to check results. Then I might look at the HTTP traffic for the functional flows. Then I might use that understanding to decide on the data coverage I need across the functional flow and as I test it with increased data coverage I will also look at the traffic and check the results in the database.
So I jump between levels of abstraction so that I can learn to improve each model as I test. Rather than fully expand each level of abstraction before moving to a deeper level.
Q: Is technical testing applicable in black-box testing?
A: I do not use the terms white box and black box testing as I find them ambiguous. I apply technical approaches to all my testing.
If I assume that by black box you mean ‘no access to the code’ then I use my technical understanding to increase the depth that I do understand the system. I might even have to decompile the application to see the code.
If I assume by black box that you mean testing driven by logical path coverage, then I will still check results and observe the system behaviour with technical skills and tools.
Q: Should you discuss your test model with developers?
A: I do. It helps me increase my technical understanding of the application, and helps them understand what I plan to do, so they can suggest new avenues of attack or different approaches.
Q: What tester would you choose for your team after graduation or with 5 years of experience?
A: I have worked with graduates, and testers of different experience levels. When I recruit I try to identify the skills and behaviours that I need on the team and the project. I assess each person in terms of how much I think they will proactively learn and how much time I will have to spend teaching them. If I evaluate that I will have to spend more time teaching them than the value they will add to the project (regardless of years of experience) then I will not recruit them.
Years of service on a CV do not equate to experience or learning. So I need different ways of evaluating, and I do this through tests and hands on testing in interviews.
Q: What if someone doesn’t have enough time for more technical testing?
A: Sometimes I have to test something very quickly. When I do this I try to maximise the amount I can learn about the application as quickly as I can – I often use my technical knowledge and tools to do that. I have to quickly make risk assessments of the functional areas and the architecture under test – I do this with my technical knowledge and tools.
Your question may stem from an environment where ‘testing’ means following scripts with an emphasis on ‘proper’ process and documentation that adds waste rather than value, and where ‘technical testing’ has the connotation of an ‘extra’ that you do after the ‘agreed’ testing has been done. I don’t work in those environments out of choice, and even when I do, I test with technical knowledge and continually expand my learning and understanding of the technologies under test.
Q: Technical test taking too much time and always after technical we should checking manual f.e. all browsers?
A: In your example of “All browsers” a technical understanding of browser differences and risks can help reduce the amount of testing you have to do on the different browsers and you may identify new ways of assessing the risk of running the application on multiple browsers.
Q: It’s hard to convince employer to create tasks of self study if you have time table and you have to register every 1 min of your work.
A: Yes, I would not find those environments conducive to learning on the job. And I don’t think I would find them conducive to effective testing either.
I was fortunate once, when recruiting for a new tester to join my team that a CV came through for a tester with no on-the-job experience for the type of testing we did or the application technology that we were testing. But on the CV they made reference to the work they were doing in their spare time, I went online and found evidence of this work. And I recruited them as a technical tester on my team. An excellent technical tester on my team. They were then able to move into an environment where they learned more on-the-job, and out of a restrictive environment that they wanted to leave.
Also I don’t restrict my learning to the time on the job. I study my profession in my free time. I understand that many people don’t do this, and have many justifications for not doing it.
I’m lucky, I remain very interested n Software and programming so I enjoy reading and learning in my spare time. By doing this I have found myself in the fortunate situation that I no longer have to work in the type of environment that you describe.
When I do exploratory testing I very often make notes on what I do every few minutes or so (or more often). I’m used to logging my work in a very detailed way. But I use simple lightweight tools and processes to do that. Your work environment may have very wasteful processes that do not contribute to effective testing and logging of your work done, or test ideas followed or generated.
You have my sympathy for finding yourself in your current job situation but how you react to this, and the steps you take to create new opportunities for yourself rests with you, and not your employer.
Q: how we can write test cases to technical test ?
A: From your technical understanding. e.g. I generate test ideas from an understanding of the architecture by looking at the flows of data and the intersection of the logical abstraction and the physical world, e.g. the flow of data between processes. I generate test ideas by looking at the content of the data in messages, looking for vulnerabilities, perceived equivalence classes (to help me restrict the scope of my testing).
Q: Slide 40 is exactly what I am doing doing so you can add also Xenu – checking all links and connections.
A: Yes I use Xenu as well.
Q: What is the best tool to use to manipulate the http headers?
A: I do this most often by using a proxy server like Fiddler or BurpSuite.
Q: What is the best way to improve your technical knowledge about testing? Books, articles, foruns, isqtb test analyst. Or do you have a better suggestion
A: Excluding the ‘ISTQB Test Analyst’, I would endorse all your suggestions: Books -I subscribe to the O’reilly safari service so that I can do a massive literature search and survey on the technologies I use. According to Google Reader I subscribe to over 670 RSS feeds, I suspect that 70% of those relate to Software topics. I don’t use forums much, but many people get value from them. I use YouTube a lot and watch many videos from conferences. I recommend the http://www.infoq.com/ InfoQ email newsletter and RSS feeds, also the QCon videos. As you subscribe to various feeds, they will link off to other areas, follow them up based on your interests. I also experiment with different tools to try and understand their capabilities.
Q: What would you suggest if you are the only tester in a project and don’t have enough time to do all testing? How would you manage the technical testing beside the functional testing?
A: Start small. Gradually increase your technical understanding of the application under test. I make my functional testing ever more technical as I learn more about the application. Gradually bring in more tools to help you with your functional testing so you gain increased ability to observe and manipulate the application.