Thanks to Xray for providing us with this blog post.
The purpose of exploratory testing is to learn how a particular area of your testing is working while using your skills as a tester to offer insightful input to your team. Through exploratory testing, you can ensure that all bugs are detected, and developers can fix them in time for the product release.
Exploratory testing is important and should be a component of your testing strategy since it helps you evaluate your tests’ efficacy, identifies code inconsistencies, and removes bottlenecks where defects are most likely to lurk.
In this post, our solution architect Sérgio Freire gives you the best tips on how to do exploratory testing in an agile environment.
Working in an Agile context
At the start of a product or one of its releases, we assume that we know everything there is to know about the product. However, there are often many unknowns and assumptions.
Agile comes as a way to deal with the complexity and unknowns around the whole software development and delivery process. When working in an Agile context, the software is delivered in small batches, known as iterations.
The idea is to reduce the batch of work and learn with small experiments. So, instead of working for a long time on a complex feature, we iterate on it by collecting feedback, learning, and incorporating our findings.
All this means that many changes are due to these iterations driven by our findings and feedback.
Exploratory Testing and Agile
The following model called the “Learning Lollipop model” (created by Sergio Freire), tries to highlight what happens during exploratory testing.
It’s a way to frame exploratory testing where we “taste” our product/product ideas (like tasting a lollipop), starting with questions that will give us ideas for designing testing experiments that we execute and then analyze. From this process, we learn. In turn, that will raise additional questions that will trigger new ideas for test experiments. While we do so, we walk in the unknown lake that contains all possible usages of our product. The more we explore, the more we find.
Using an example, let’s first try to see how these actually work well together. Say we are working in a system with a set of features, aiming to add a new feature using an Agile approach.
From a testing perspective, what we had (i.e., how the system behaved) should be covered by a set of checks, using test automation scripts as much as possible. This will allow us to collect almost immediate feedback from the CI pipeline(s).
We cannot simply retest everything from the past “by hand.” We also know that test automation scripts are fallible because they will always be limited to testing what they’re hardcoded to check; however, they give us a good starting point.
Whenever iterating on a new feature, we know that we don’t know much about it beforehand; that’s why we’re iterating it, after all. Usually, we’re dealing with a rough user story and not an extensive, highly detailed requirement.
Therefore, we need to test our initial ideas for the user story and depict areas/risks we should have in mind. Many questions will come, at the start, during, and after the implementation. All these can become ideas for test charters that we can explore with Exploratory Testing sessions.
Remember that in an Agile context, changes are frequent, and risks also change very dynamically.
Exploratory Testing is a great fit in Agile, as it is extremely flexible and doesn’t require upfront preparation (as happens with manual scripted test cases). It also uses information from previous sessions to drive new testing sessions. Therefore, it adapts to changes as it doesn’t assume a certain state and expected results for the system.
Tips for exploratory testing
Exploratory testing mockups.
Perform exploratory testing sessions on early mockups, internally and with users. This can be quite helpful to optimize flow problems, for example, and highlight the most valuable ones. You can also apply exploratory testing during your design sprints.
Discuss upfront with the team possible charters for your exploratory testing session.
During regular meetings (e.g., standups, planning), discuss with the team the test charters (i.e., the questions you aim to answer during testing). It’s a good moment to talk about risks and have insights from different team members, giving ideas for further exploration.
It’s always a good time to perform an exploratory testing session.
Whether you’re adopting waterfall or Agile, it’s always a good moment to perform some exploratory testing. We will never know everything about our product/system and its context, but we can further improve our understanding by conducting exploratory testing sessions. There are many quality attributes we can look at, for example. Consider aspects that concern your team, users, and business, and use that to drive new sessions. Taking some time to explore is investing in knowledge so that we can then work towards incorporating that feedback and improving our product/system.
Use exploratory testing to highlight ideas for test automation scripts.
Features should come with code, including unit and integration level tests and even system tests if appropriate. Whenever performing exploratory testing, one of the outputs can be ideas for test automation scripts. During exploratory testing, we may find flows, impacts, and edge cases, for example, that, due to their relevance, should be covered by “automated tests.”
This applies to the waterfall and Agile projects and will allow us to improve test coverage addressed by test automation and hopefully gain more time to focus on other tasks (e.g., further exploration, fixing problems, etc.).
Perform exploratory testing on the feature branches or the PRs.
If your team uses feature branches while features are being implemented, you can and should test. This means working with developers to improve the feature iteratively. You may perform an exploratory testing session around a certain risk, quality attribute, or subset of that feature at a given moment. You can also perform a session when the PR is ready for review; if you tested while it was implemented, then this moment will eventually be more of a high-level type of charter.
Perform exploratory testing after merging branches.
Merges sometimes produce unexpected results. Even though the feature branch may (should) include automated tests, there can be unexpected consequences, so scheduling an exploratory testing session can help uncover them.
Involve developers and other roles in exploratory testing.
Besides testers, getting others on board for exploratory testing can provide additional perspective. At the same time, foster a quality centric culture where team members can improve quality from the start in the future.
Pairing with a developer, with the PO, with a designer is a good practice to understand not just the system from different angles but also what different stakeholders expect from it; besides, it’s an excellent mid/long-term investment towards better quality.
Don’t limit exploratory testing to non-regression.
Even whenever automated regression tests may cover existing features, it’s a good practice to perform exploratory testing also for regression testing if you have the opportunity to. Test automation can cover the essentials, but we know many things escape these tests as they will always be limited in number and scope. Looking back at your previous features with your eyes wide open may depict problems added meanwhile and problems that you didn’t have the opportunity to uncover.
Exploratory test your test automation.
Look at your existing test automation and explore it to look for problems (it’s also code, isn’t it?). Look also for problems in scope, concurrency, and relevance. Look at your existing test automation logs, as they may provide valuable information or expose too much or too little information.
Exploratory test using tools to augment testing and gain efficiency.
Tools are used to perform certain tasks with efficiency and consistency. In exploratory testing, tools are used to augment the tester’s capabilities, not to replace the tester. An exploratory tester will easily use tools to facilitate API requests and assist with performance testing. With tools, an exploratory tester can be more efficient and cover quality aspects that otherwise would be hard or even impossible to tackle.
Exploratory test looking for gaps and opportunities to improve the value.
While testing, we look for problems that affect the quality and, therefore, the value as seen by different stakeholders. Testing is about understanding how the system works connected with expectations from all these different stakeholders. In this sense, testing is also about finding opportunities to increase the value. During exploratory testing, and using our knowledge and background, we can depict ways of improving the value of our products. Maybe that can be about framing the feature slightly differently, trying out a new form or interaction.
Bringing some agility with Exploratory Testing to waterfall projects
For organizations working on waterfall-based projects, testing mostly occurs after features have already been implemented. We know that if this happens, then the cost of fixing problems increases considerably.
Usually, there are initial requirements that drive implementation. These highly detailed specifications are not immune to problems; on the contrary: they can be built on top of many assumptions and lack actual user feedback.
We know that requirements, and specifications, in general, are incomplete, ambiguous, sometimes contradictory, and easily get outdated.
As exploratory testers, we can use not only requirements and other documents as the source for our tests; we also understand the context of our product, know about similar products, and know of known heuristics that can help us expose problems through test tracking and reporting. We also have our background that we can use to expose risks and impacts that otherwise could escape traditional testing.
In waterfall projects, we can use Exploratory Testing to help us:
- Uncover problems, risks, and gaps that we couldn’t predict beforehand as they were not identified in the requirements/specifications.
- Introduce testing while the feature is being implemented and thus refine it before it’s too late.
- Complement traditional approaches, such as manual scripted test cases, with exploratory testing to go beyond the obvious and expose problems that we could otherwise miss.
Unleash your testing potential with Exploratory Testing
Exploratory testing promotes innovation instead of scripted testing that is centered around specified test cases and attempting to complete a fixed number of tests per day. Exploratory testing encourages us to act role play as the end-user and detects more realistic bugs.
Exploratory testing is highly helpful in agile environments and has several advantages, as seen above. QA teams can successfully use this testing strategy for their own success in the agile development process by knowing its benefits and using reliable test management software like the Xray Exploratory App.
Author
Sérgio Freire, Head of Solution Architecture & Testing Advocacy at Xray
Sergio Freire is a solution architect and testing advocate, working closely with many teams worldwide from distinct yet highly demanding sectors (Automotive, Health, and Telco among others) to help them achieve great, high-quality, testable products.
By understanding how organizations work, their needs, context and background, processes and quality can be improved, while development and testing can “merge” towards a common goal: provide the best product that stakeholders need.
Xray is an EXPO Platinum partner at EuroSTAR 2023, join us in Antwerp