Let’s be honest. The dream of 100 percent test automation often turns out to be a nightmare. After all, it’s not just about writing a few scripts. Successful test automation needs to be well thought out, requires a test architecture and, above all, time. If you let the reins slip for even a few iterations, technical debt will creep in. In the worst case, the test cases become flaky. 

Less spectacular, but just as painful in the long run, is gentle erosion. While the system to be tested is constantly evolving, the automated scripts lag behind. They may still run successfully, but over time they become less and less meaningful.

“Not true!”, you are probably thinking. “We do test-driven development.” That’s laudable, but probably only part of the truth. Because you certainly have them too – the functional manual testers who cover the upper part of the test pyramid and look at the system as a whole, perhaps even in interaction with other applications. 

The Importance of Functional Testing

Functional testing is important. Of course, as much as possible should be checked during unit testing, but to be honest: the nasty bugs are hidden between the components. An example:

An app requires 2-factor authentication for login. Therefore, an 8-digit code consisting of letters and numbers is sent by email. Unfortunately, the app only allows numbers to be entered.

This example is not made up and illustrates the difference between unit and system testing. Each component complied with its specification and was tested successfully. Nevertheless, the overall system is unusable because the error was not in the code, but in the specification. Functional testers find such errors and that is what makes them so valuable.

Manual testing has never been easy, but today, in the age of Agile, it’s a pure race against time. If they don’t want to be left behind, they need to understand the intricacies of automated functional testing. Because test automation relieves them of the repetitive regression tests and allows them to focus on the important things, such as new features.

Fundamental Principles of Effective Functional Test Design

In fact, there are several common principles that are important in both manual testing and test automation:

• Clarity : Clarity is key. Structured, easy-to-grasp tests improve comprehension and minimize ambiguity, benefiting all stakeholders. Visual diagrams have long helped developers simplify complex problems. Testing can benefit from similar visualization, as functional tests mirror the intricate nature of systems under test. Visual test design definitely improves clarity, making it easier for test automation engineers to understand the business aspects of functional testing.

• Modularity : By breaking down complex scenarios into manageable test cases, manual testers can lay the groundwork for seamless automation, ensuring that each test remains a valuable asset throughout the software development lifecycle.
  
• Maintainability : Functional tests will continue to evolve, as will the associated test scripts. Some changes affect the technical level, others the functional level. Keyword-driven testing is a proven method of separating these two levels. Manual testers can thus contribute to the maintenance of automated tests without having to program themselves.

The goal of an effective, functional test design must therefore be to develop tests that are easy to maintain and update.

Transitioning from Manual to Automated Execution

Automation is often viewed as process optimization, as it takes over error-prone, repetitive tasks. Testers can use the time gained to focus on those tasks that require human judgment. Executing existing tests repeatedly following the accelerating rhythm of testing cycles is certainly not a mentally demanding task. The dumb writing down of test procedures is not a great intellectual accomplishment either. The real value lies in the preliminary considerations: What situations could occur? How should the system behave in this case? What can I do to push the system beyond its limits? Automation is therefore also a helpful and welcome support when creating test cases.

Studies[1] show that the combination of intellectual performance in test design and automated test execution not only increases test speed, but also the coverage of requirements and code. In general, the reliability of tests is increased when automated scripts support manual testers, especially in lengthy tests. This can be seen very clearly in load and performance tests, which are unthinkable without automation.

An intelligent automation strategy effectively balances human expertise with automated tasks.

A Practical Case Study – Yest Augmented by Maveryx

In this example, we show the successful marriage of two concepts represented by Yest and Maveryx.

Yest is a visual test design tool that implements a modern form of model-based testing (MBT) and test generation. Yest offers a whole range of functions to enhance the creative work of test design and speed up the painstaking task of writing test cases. Yest itself is agnostic, meaning that the generated scenarios may be used both for manual and/or automated test execution.

Maveryx is an automated software testing tool that provides functional UI, regression, data-driven and codeless testing capabilities for a wide range of desktop and Web technologies. Its innovative and intelligent technology inspects the application’s UI at runtime as a senior tester does. With Maveryx, there is no need for code instrumentation, GUI capture, maps and object repositories.

From Visual Test Design for Manual Tests…

With Yest Augmented by Maveryx, the functional manual testers concentrate on creating graphical workflows and define actions including the expected results for the various cases to be tested. Yest then generates test scenarios from these workflows and the stored information, which can be executed directly manually.

…to Automated On-The-Fly Test Execution

But can they also be automated and, if so, what results can we expect? This is where Maveryx comes into play. All you have to do is to provide a detailed instruction of manual steps in Yest. Yest Augmented by Maveryx recognizes these instructions for manual execution (for example “Click on Submit”) and executes them automatically.

If a step is not interpretable or fails, Yest Augmented by Maveryx stops and waits for manual input. You may then abort the test or do the necessary steps manually and continue. The execution results are reported in Yest Augmented by Maveryx or in the test management you read the test cases from.

Use Cases

Yest Augmented by Maveryx serves in various situations:

1. Hardening the test procedure : Manual tester may execute their tests without having to code a single line of code. This provides them with rapid feedback on the quality of their tests. No more tedious analyses and hot disputes about whose fault it was.
2. Executing manual tests directly from Xray or MS Excel : During official test runs, manual testers can call upon Yest Augmented by Maveryx to do part of the work. In fact, it is possible to import tests from Xray or MS Excel and execute them even without having used Yest Augmented by Maveryx for test design.

As functional manual testers embark on their automation journey, they will gain valuable insights into creating robust, maintainable test suites that stand the test of time.

Conclusion

This blog aims to guide manual functional testers through the complicated process of developing functional tests and harnessing the transformative power of automation. In an age where software development demands agility and speed, understanding the intricacies of automated functional testing is critical for testers looking to optimize their workflows.

Clarity, modularity and maintainability are key success factors for a successful transition from manual to automated test execution. Visual test design and model-based test case generation pave the way to structured tests and complete coverage. With the right tool support, it is possible to carry out these tests automatically without having to write a single line of code. Yest Augmented by Maveryx provides this support. Contact us to learn more at [email protected] or [email protected]

[1] Khankhoje, Rohit. (2023). Revealing the Foundations: The Strategic Influence of Test Design in Automation. International Journal of Computer Science and Information Technology. 15. 10.5121/ijcsit.2023.15604.

Authors

Alfonso Nocella, Co-founder and Sr. Software Engineer at Maveryx 

Alfonso led the design and development of some core components of the Maveryx automated testing tool. He collaborated in some astrophysics IT research projects with the University of Napoli Federico II and the Italian national astrophysics research institute (INAF). Over the decades, Alfonso worked on many industrial and research projects in different business fields and partnerships. Also, he was a speaker at several conferences and universities.

Today, Alfonso supports critical QA projects of some Maveryx customers in the defense and public health fields. Besides, he is a test automation trainer, and he takes care of the communication and the technical marketing of Maveryx.

Anne Kramer, Trainer and Global CSM at Smartesting

Anne Kramer first came into contact with model-based test design in 2006. Since then she has been burning for the topic. Among other things, she was co-author of the “ISTQB FL Model-Based Tester” curriculum and lead author of the English-language textbook “Model-Based Testing Essentials” (Wiley, 2016).

After many years of working as a process consultant, project manager and trainer, Anne joined the French tool manufacturer Smartesting in April 2022. Today, she is fully dedicated to using models for testing purposes. This includes visual test design and, more recently, generative AI.

Maveryx is an Exhibitor at EuroSTAR 2024, join us in Stockholm.

End-to-end testing is one of the most effective ways software teams can understand the customer experience. Unlike unit or component testing, which focus on individual pieces of the application, E2E tests seek to understand product quality as an integrated journey. In many ways, end-to-end testing exemplifies the expanding role of software testing in a DevOps world: the crucial connection between how software is built and how it is used. When development teams understand how their changes will impact their end users, they’re better able to deliver value to those customers. When automated effectively, end-to-end testing provides this connection quickly enough to support continuous delivery.

But the shift to digital-first experiences means that end-to-end testing needs to evolve and expand, running contrary to established testing best practices. Even a simple user journey, such as the one outlined below, likely involves multiple third-party APIs and services as well as email touchpoints and personalized offers or recommendations. Development teams must build seamless user experiences that make a complex customer journey feel simple. Expanding the definition of end-to-end testing ensures they can do so successfully. But with traditional testing frameworks, complicated automated testing meant a high risk for broken tests, extra maintenance, and inaccurate results, which ultimately slowed down development pipelines. Quality teams instead opted for shorter, simpler end-to-end tests that were less likely to break as the product evolved. But the holistic view provided by true end-to-end testing is extremely valuable – if quality teams have the tools to manage them.

New Customer Journeys Demand Broader End-to-End Testing

The image above includes an example of an end-to-end test for an ecommerce website. Despite this being a fairly simple – and common – transaction, an automated test needs to cover a marketing email, a coupon code, and an invoice email with a PDF attachment. But the story doesn’t end there: it’s extremely likely that the checkout test step includes an API for a payment service like Square or Afterpay. It’s also likely that coupon codes are personalized for customers, given that loyalty programs with customized rewards are proven to increase consumer spending.

Skip these steps, and there is a real risk to revenue. If a marketing email fails to accurately show a customer’s coupon code, conversion rates will suffer, impacting sales and potentially churning previously loyal customers. Managing this type of comprehensive test is essential for supporting quality customer experiences.

The Challenges of Comprehensive End-to-End Testing

Though the above end-to-end test is critical for understanding the user experience and how each change will impact it, such tests pose several challenges for developers and software testers. First, maintaining such an extensive automated test with scripted test automation frameworks is likely to consume a significant amount of a testing team’s time and effort, which has a serious impact on an organization’s ability to accelerate product velocity. Since additional test steps increase the risk of a test breaking, most quality teams avoid creating longer end-to-end tests in order to reduce the burden of test maintenance. But what they gain in reliability, they lose in test coverage.

Second, comprehensive automated tests often require longer investigations into failures. Combing through a long list of test steps to identify the specific cause of a test failure can take valuable hours, a luxury development teams don’t have as delivery cycles shorten. Considering that 44% of developers say that investigating failed tests is a significant pain point, quality teams must have effective strategies in place to triage comprehensive end-to-end tests when necessary.

Maintaining More Complex End-to-End Tests

An end-to-end test covering email, API, and non-functional test steps is highly susceptible to any product changes, but advances in AI and machine learning have reduced the amount of time and effort needed to maintain automated tests, making it possible for quality teams to manage comprehensive end-to-end tests. Using unique identifying elements across an application’s UI, including shadow DOM components, intelligent test automation solutions can detect product changes and update end-to-end tests accordingly.

Automating end-to-end test maintenance not only ensures that test maintenance is less labor-intensive, but also allows more team members to contribute. For example, manual testers can more easily collaborate on E2E tests that contain integrated API tests, ensuring that comprehensive end-to-end tests capture the full user journey and accurately assess quality.

Reporting on End-to-End Testing

Even when end-to-end tests are maintained, identifying the root cause of an error can be time-consuming, causing delays and disruptions in the later stages of the SDLC. Rapid results that support fast bug resolution is critical for delivering exceptional user experiences at the speed of DevOps.

Advances in cloud-based testing and the availability of SaaS test automation tools are making it easier to scale and maintain comprehensive end-to-end testing strategies. Cloud-based runs give in-depth insights that support continuous improvement, and can be run on a schedule or as part of a CI/CD pipeline. Flexible execution options make it possible to routinely and reliably run comprehensive end-to-end tests without slowing development. But perhaps even more importantly, integrating end-to-end test automation into existing development workflows allows developers to quickly act on end-to-end test results.

Building workflows that surface comprehensive end-to-end test results in a digestible way supercharges their value. Sharing test results as Jira tickets, complete with screenshots of the point of failure, DOM snapshots, and performance logs, is ideal for triaging comprehensive end-to-end tests since developers can easily identify what test step caused the failure. The time from failure to fix becomes much shorter, making comprehensive end-to-end tests highly actionable.

The Future of End-to-End Testing

comprehensive end-to-end tests are often considered too time-consuming to provide real value to development teams. But their ability to ensure quality of the perspective of the customer is invaluable, even essential, in a time where every business is competing on their digital customer experience. Overcoming test maintenance, execution, and investigation obstacles to comprehensive end-to-end tests gives development organizations a powerful tool for understanding how changes will impact their users. And with the right test automation solution, end-to-end testing becomes an adaptable process that can continuously evolve to match real customer needs. A few examples include automated accessibility checks, integrated API tests, shadow DOM components, and cross browser testing. No matter what your customers need, comprehensive end-to-end tests will help your team deliver exceptional user experiences. 

Author

Bridget Hughes., Content Marketing Manager at mabl

Bridget is the Content Marketing Manager at mabl, the unified test automation platform for delivering modern software quality. She’s dedicated to helping quality teams expand testing and improve product quality through educational blogs, articles, and the occasional software testing meme.

Mabl is an Exhibitor at EuroSTAR 2024, join us in Stockholm.

Behavior Driven Development (BDD) is a well-regarded way to write application requirements as scenarios that describe the behaviour in various contexts. BDD evolved from the agile movement and its emphasis on Test-Driven Development (TDD) to take things one step further than simple user stories and document a user’s behaviour when they use a system.

Typically using the Gherkin syntax, a user scenario in BDD is written the following way:

• Given: the initial context at the beginning of the scenario, in one or more clauses;
• when: the event that triggers the scenario;
• then: the expected outcome, in one or more clauses.

Gherkin is a business readable language that helps you to describe business behavior without going into details of implementation. It is a domain-specific language for defining tests in a standardized format for specifications. It uses plain language to describe use cases and allows users to remove logic details from behavior tests. For example:

The primary benefit of BDD is that it encourages communication between developers and other stakeholders, such as product owners and users. BDD helps bridge the gap between technical and non-technical stakeholders by providing a common language for discussing the behavior of the system. By using this language, stakeholders can understand each other’s needs and expectations, leading to better development decisions.

In addition to being a useful way of describing a requirement specification, the text in the Gherkin language acts as both documentation and the skeleton of your automated tests. For example, test automation engineers often take the Gherkin scenarios and use a framework like Cypress or Robot Framework to turn these high-level user interactions into executable test automation scripts. However, this process is manual and can be time consuming, with the automation engineers having to hand-write large amounts of Python or JavaScript to turn one BDD scenario into a functioning automated test.

Enter the Power of Generative AI

With Generative Artificial Intelligence (GenAI), you can use the power of Large Language Models (LLM) to automate a lot of this process. Currently, our SpiraPlan quality and test management system uses GenAI to automatically generate BDD scenarios, test cases, and risks from simple agile user stories:

Figure 1: BDD Gherkin Scenario Generated by AI.

However, this is just the beginning of what will soon be possible!

Using the latest LLMs such as GPT4, we can pass in the BDD scenario text as a prompt to the LLM and it will generate a set of page objects and associated page object models functions/methods. This means that a simple human-readable scenario can automatically turn into a Selenium-style set of page object model function calls.

Figure 2: A Human readable scenario

Figure 3: An automated test script using page objects, automatically generated by AI.

Finally, when you feed either in a specially tagged image of the application or a reduced version of the page DOM (to avoid using too many GenAI tokens), the LLM is able to implement each of the page object model functions into the appropriate code to interact with the application and test its user interface. These could be either image-based clicks or WebDriver-style CSS selectors depending on what you used to prompt the model.

This means we are close to having the holy grail of taking a BDD scenario and automatically converting it into an 80-90% ready-to-run automated testing script.

What is the Role of Testers?

As we often say, GenAI is not here to replace humans, but instead to assist. If we consider this new Human-AI team, the human testers’ job is to create/review scenarios from the AI, review and optimize draft automation code from the AI, and look for weaknesses, edge conditions, and missing cases. Working together, the Human-AI team will be able to reduce higher-quality applications in a faster time than ever thought possible!

Author

Adam Sandman, Director of Technology at Inflectra

Adam Sandman has been a programmer since the age of 10 and has been working in the IT industry for the past 20 years in areas such as architecture, agile development, testing, and project management. Currently, Adam is a Director of Technology at Inflectra, where he is interested in technology, business, and enabling people to follow their passions. At Inflectra, Adam has been responsible for researching the tools, technologies, and processes in the software testing and quality assurance space. Adam has previously spoken at STARWEST, Agile + DevOps West, STPCon, Swiss DevOps Fusion, InflectraCon, TestingMind, EuroSTAR, Agile Testing Days, and STARCANADA.

Inflectra is an Exhibitor at EuroSTAR 2024, join us in Stockholm.

Why AI Needs a Multimodal Test Strategy!

It seems as though every organization is infusing artificial intelligence (AI) into its applications and processes these days. And why wouldn’t they? Generative AI and large language models (LLMs) have made it easy to integrate AI and machine learning (ML) capabilities into digital experiences, tools, frameworks, platforms, and more.

Some of these AI-based systems are now multimodal, meaning they combine different forms of data to make predictions and draw insights about real-world problems. Multimodal AI trains on and utilizes a combination of images, video, text, speech, audio or numerical data. For example, GPT-4 Turbo makes it possible to feed an image to the model with a text prompt, or request that visuals be used to support an explanatory response.

Beyond the context of AI (pun intended!), multimodality simply refers to activities having different modes. As such, there are many things that can be described as multimodal. Software testing, for example, is multimodal because there are several types of testing, each with formal and informal techniques, methods, and supporting tools. However, for many years the testing industry has been focused on what I consider to be a single mode of testing — pre-production testing. That is, testing that occurs prior to the release of a system or component.

Proponents of the shift-left testing mindset, including myself, advocate for paying attention to quality as early as possible in the software development lifecycle to reduce the cost and impact of late or escaped defects.

For AI-based systems, a pre-release testing strategy that emphasizes shifting left is important. However, because of its dynamic nature, it is equally important to test AI on the right, post-release, commonly referred to as production testing. This is the first article in a two-part series that describes a multimodal testing strategy for AI based on one of our mantra’s at Test IO — Shift-Left, Test Right!

Shifting AI to the Left

So what does a shift-left, pre-production testing model for AI systems look like?

It starts with understanding that a good testing strategy for AI must be holistic, and include the perspectives of product, engineering, data science, testing, performance, security, operations and the end user community.

A Holistic Quality Perspective for AI

With key stakeholders at the table, you can now begin applying a systematic, disciplined, and quantifiable approach to AI software development.

Foundational AI Engineering Practices

In 2019, the Software Engineering Institute (SEI) of Carnegie Mellon University published a set of Foundational Practices for AI Engineering. Here are a few of them that I paraphrased and can attest to have lived out in a former life as a Chief AI Scientist:

• AI Problem First, Then Solution — Leveraging AI may often result in a more complex and expensive solution than a non-AI alternative, and therefore it is important to start with a well-defined problem that requires an AI solution. On the upside, the resulting AI solution may end up being more robust so it is important to weigh all of these factors into your decision.
• Choose Algorithms Based on Needs, Not Popularity — Select algorithms based on their appropriateness for solving the problem being tackled, not on their popularity. AI and ML algorithms may differ in the kinds of problems they address, level of detail of the output, interpretability, and robustness. Avoid using algorithms as “shiny new toys” and only change them to meet the needs as the system evolves or to adapt it to a new environment.
• Rule Your Data, Or Be Ruled By It — The output of an AI system is generally tied to the data used to train it. If you don’t take your data seriously it can consume the entire project. Make sure you allocate sufficient time and effort for managing your data, accounting for the need for data ingestion, cleansing, protection, validation, testing, and monitoring.
• Design AI for Ethics and Security  — While data collection often raises questions concerning the privacy of AI, there are other ethical issues that should be addressed early during development. These include both fairness in data representation and decision-making based on diversity factors such as age, gender, ethnicity, disability, and more.
• Implement Flexible, Extensible Solutions — The boundaries of components in an AI-based system are more sensitive to change than those in traditional software. This is because data dependencies in AI systems may trigger changes in the expected outputs, system functionality, and infrastructure. It is therefore imperative to implement loosely coupled, flexible solutions to be able to keep pace with inevitable changes in the data, models, and algorithms.

To ensure that the aforementioned AI engineering practices are followed and implemented correctly, there should be validation and verification at each stage of the development process. In other words, the notion of continuous testing must be applied to AI software development.

Continuous Testing During AI Development

Applying continuous testing to AI development may require bringing a testing mindset to the stakeholders involved in development and/or having testers wear multiple AI development-related “hats”. Aspects of AI development that can be validated early as part of shift-left include testing the training process and testing the training data.

Testing the Training Process

Data scientists or AI/ML engineers are typically responsible for training the system. Although the strategy for continuous testing will depend on the specific type of machine learning being used, there is a general concern that should be addressed:

Model Fitting — during the training process, the selected model is fit on the training data in order to make predictions. In ML, fit refers to how well the model is able to approximate a given function. As shown in the figure, a model may be underfit, overfit or have a balanced fit.

• Underfitting is when the model has not been trained enough and therefore misses the trends in the training data. This can happen if you have too little data or a very simple model.
• Overfitting is when the model how has been trained too much and therefore matches too closely to the training set. In such cases, the model may be capturing noise in the data instead of, or in addition to, the underlying data pattern. This can happen when the model trains too long or is too complex.
• Balanced Fitting is when the model has been trained “just right” and there it is a good approximation of the true function, which represents a solution to the problem under investigation.

Understanding the notion of model fitting repreents the tip of the iceberg when it comes to validating the training process. However, it should be clear that if testers are going to contribute to this aspect of shift-left, they will have to ramp up on data science and AI/ML engineering. Unlike the training process, data is an area of AI/ML that testers are likely to be able to add value to with little or no lead time.

Testing the Training Data

Software testers deal with data all the time, and although it may not be for the same purpose, many of the issues encountered with training data are similar to those for test data. Furthermore, a common technique for testing ML systems during development involves partitioning the dataset into a training set and a test set. Therefore, some of the data problems related to AI/ML development, directly correlate with the test data selection and test data management issues. These include, but are not limited to:

• Duplicate Data
• Outdated Data
• Incorrect Data
• Incomplete Data
• Inconsistent Data
• Insecure Data
• Non-Representative Data

The testing mindset provides on added superpower when it comes to data — the end user’s perspective. User empathy is commonly cited as one of the traits of a good tester. It enables one of the most important activities surrounding the validation of training data — assessing AI fairness.

One of the key goals of evaluating AI fairness is to answer the question: Are there groups of people who are disproportionately, negatively affected by the system. A quick AI fairness-related check is to inspect the sample size distribution of the input features and/or outcomes, especially for sensitive attributes like age, gender, race, among others.

Assessing AI fairness goes way beyond spot checking sample size distributions. In fact, most of the tech giants are associated with the development of tools and frameworks to support testing AI applications for fairness issues. These include Fairlearn, AI Fairness 360, and the What-If Tool.

Summary

This article explored the concept of shift-left testing as it may be applied to AI systems. Shifting AI to the left requires thinking holistically about testing these types of systems, and rooting their development in foundational engineering practices. Validation of the training process and training data, among other activities, should be integral to AI development. However, this type of pre-production testing for AI only represents a single-mode of activities.

Stay Tuned.

Author

Tariq King, CEO and Head of Test IO

Tariq King is a recognized thought-leader in software testing, engineering, DevOps, and AI/ML. He is currently the CEO and Head of Test IO, an EPAM company. Tariq has over fifteen years’ professional experience in the software industry, and has formerly held positions including VP of Product-Services, Chief Scientist, Head of Quality, Quality Engineering Director, Software Engineering Manager, and Principal Architect. He holds Ph.D. and M.S. degrees in Computer Science from Florida International University, and a B.S. in Computer Science from Florida Tech. He has published over 40 research articles in peer-reviewed IEEE and ACM journals, conferences, and workshops, and has written book chapters and technical reports for Springer, O’Reilly, Capgemini, Sogeti, IGI Global, and more. Tariq has been an international keynote speaker and trainer at leading software conferences in industry and academia, and serves on multiple conference boards and program committees.

Outside of work, Tariq is an electric car enthusiast who enjoys playing video games and traveling the world with his wife and kids. 

EPAM is an Exhibitor at EuroSTAR 2024, join us in Stockholm.