Thanks to Anne Kramer, Global CSM at Smartesting for providing us with this blog post.
Why did the smart tester refuse to use manual testing methods? Because they didn’t want to be called “dumb testers”!
This joke created by ChatGPT sums up well the prevailing spirit in software testing. To reach the goals, which are better products, cheaper tests and faster delivery, most organizations rely on test automation. But is this smart?
Yes, it is. However, it’s anything but smart to drive very fast with your eyes closed and that’s exactly what happens in many cases. People are automating come hell or high water, without really questioning how useful the tests actually are. In the end, automated tests can be repeated ad infinitum, and we are surprised that they still miss numerous defects. We are on the racetrack, but we are going in circles.
Similar to a car trip, test automation also needs a plan where to go. Test automation requires smart test design.
Shift Left!
A commonly accepted way to strengthen QA in general and test design in particular is Shift-Left Testing. Today, testing early and often is generally considered as being a good way to reduce risks, costs and to improve quality. The earliest possible time to consider testing is at the beginning of development, when requirements are defined.
This is how Acceptance Test-Driven Development (ATDD) came up. Test scenarios serve as specification by example and replace detailed requirements descriptions. Behavior-Driven Development goes one step further and facilitates automation through the semi-formal Gherkin syntax. The user story’s acceptance criteria are directly described through acceptance test scenarios.
Visual ATDD
But acceptance testing is far more than just acceptance of individual user stories. Functional system testing should verify that the system under test can be used in a larger context. It is important to check entire workflows, thus covering several user stories in a row. Here, the semi-formal Gherkin syntax reaches its limits. Long scenarios are difficult to understand and even more difficult to keep up-to-date.
To solve the problem, we have to design our system tests incrementally, following a test-driven, top-down approach. To keep the overview and to cope with complexity, visual representations are extremely helpful. If we add to that the principles of Model-Based Testing (MBT), we get Visual ATDD, a lean and agile version of MBT.
Yest® by Smartesting
In Visual ATDD, system and end-to-end (E2E) tests are specified graphically and grow incrementally from sprint to sprint. With Yest®, Smartesting’s visual test design tool, you may also add the business rules in tables linked to the graphical workflows. Yest® then generates manual or automated tests and publishes them into the test tool of your choice. A large number of accelerators facilitate the maintenance of existing tests to keep them compliant with potentially modified workflows.
Shift Right!
But shift-left is not the only way to smarter test automation. Since the beginnings of Agile, the wheel has continued to spin. The DevOps idea came up and with it the maxim “Shift-Right”. Why should we stop testing with delivery? Let us perform tests and evaluate quality under real world conditions. Continuous deployment pushes this concept to the extreme. Once a new feature passes the quality gate composed of automated regression tests, it is released, but it may be rolled back immediately if problems are observed in the field.
Testing in production has many advantages. By monitoring an application in its users’ hands, we can quickly discover errors, bugs, and performance issues and fix them before they cause any more damage to our customer’s satisfaction. We can also test two versions of the design of a feature to see which one works better for our users (A/B testing) or even learn how our users appropriate our application.
Usage-Centric Testing
Usage-centric Testing complements the shift-left approach by using the knowledge of user’s behavior to design more accurate tests. In particular, it allows us to define targeted regression test suites covering relevant usage scenarios from end to end – targeted, because they cover what really users actually do.
In fact, when writing acceptance tests, we test how we think our users are going to interact with a new feature. In best cases, the feature has been designed in the most user-centered way possible. But even so, our end-users are… human beings. And as human beings, their behavior is not predictable and they tend to appropriate our products in ways we weren’t expecting. For example, we never expected smartphone owners to use their expensive high-tech device as a flashlight. If we never validate our assumptions, we take the risk of keeping irrelevant E2E tests that we still have to run and maintain
Gravity by Smartesting
Gravity is Smartesting’s solution usage-centric testing platform. It collects anonymized usage data from web applications and analyzes them using machine learning techniques. The software detects patterns in users’ behavior and compares test coverage of those patterns obtained in the development environment against product usage. In the end, it allows you to generate automated regression tests using Cypress. Thus, you obtain a regression test suite which perfectly reflects the actual usage of your application.
Perform the Split!
Shift-left or shift-right is not an either/or decision. They are two sides of the same coin. Both aim at creating smart tests; SMART in the sense of the well-known acronym for five quality characteristics: Specific, Measurable, Attainable, Relevant and Timely.
With Yest® you obtain few, but highly relevant tests that specifically cover the workflows and business rules you collectively wish to check. Traceability and coverage indicators provide quantitative measures of how good your tests are. Thanks to the numerous accelerators, the tests may be specified incrementally and sprint goals are attained. Shared in Jira and, on top of that visual, the graphical workflows represent a first validation of the expressed requirements. You get timely feedback from all kinds of stakeholders.
Gravity, on the other hand, concentrates specifically on the relevance of E2E tests. It hardens your regression tests by using information from productive systems. In a DevOps approach, Gravity allows you to monitor usages and to detect deviations timely. It also measures the coverage of real-world usages through the existing test suites and generates relevant automated test scripts to increase this coverage to achieve right testing.
YEST® is a visual-based tool for designing and automating functional tests in Agile.
By accelerating the production of business-relevant tests, YEST® significantly and measurably reduces the costs of creating and maintaining manual and automated test cases (30% for design and 50% for maintenance).
The free YEST® for Jira add-on allows teams to share and capitalize on their functional knowledge via graphs and forms the basis for collaboration between business teams, QA teams and software developers. Interested? Contact us for a demo.
Gravity helps Agile/DevOps teams to deliver high quality software faster.
Dedicated insights and AI will enable them to design and dimension their E2E test suites to efficiently cover the real application use in production. You can try Gravity for free at https://www.gravity-testing.com/
Author
Anne Kramer, Global CSM – Smartesting Solutions & Services, Germany
Anne is an expert in test design approaches based on visual representations and works as Global Customer Success Manager at Smartesting.
Smartesting is an EXPO Exhibitor at EuroSTAR 2023, join us in Antwerp