Thanks to Micro Focus for providing us with this blog post:
The constant surge in digital transformation forces organizations to perform tests on an increasing number of platforms – but still get results as quickly as possible. Manual testing just can’t keep up with the speed of business, so teams turn to test automation. But traditional software testing tools are only effective to a point. Generally, these tools rely on identifying objects on the screen through their internal representation—e.g. coordinates, class name, type, and many other. This method of identifying objects can be very fragile. Even a small change might result in the tool failing to find the object. The drawbacks of these techniques prevent teams from scaling their test automation efforts up to the levels they require.
To that end, the most common test automation challenges include:
- Relentless test maintenance —tests that rely on unique object properties can be susceptible to breaks, thereby making testers perform regular updates to ensure their tests still run on each supported environment.
- Test execution time is too long — even if a test set runs without interruption, it can take a significant amount of time to run all the tests to completion.
- Insufficient test coverage — teams must support an ever-expanding range of platforms, devices, and operating systems, requiring testers to customize the tests for each environment.
- Test creation fatigue — it takes time to build and design effective tests, with much effort required to uniquely identify on-screen objects that are part of the test.
What our research at Micro Focus revealed was that automated object detection with computer vision is key to lowering these barriers.
Computer vision-based AI for automated object detection
Recognizing objects without knowledge of their internal representation was one key objective when we developed our AI engine. This goal was accomplished by combining AI-based algorithms that accurately and consistently recognize objects regardless of device or environment.
For example, a test step might require clicking the shopping cart icon on a mobile app. The AI engine should be able to locate the shopping cart icon on the current screen without knowing:
- If the screen is on a mobile device.
- Whether the device is running Android or iOS.
- If the screen is a desktop browser.
- Whether it’s Chrome, Firefox, Edge, or another browser.
The ability to “Click the shopping cart” step should work under any circumstance with an AI engine. At Micro Focus, we have successfully implemented this in the Micro Focus UFT Family of software test automation products, and the AI techniques used in the engine include computer vision through an artificial neural network and optical character recognition.
Defining computer vision
An AI engine understands a screen’s composition and breaks it down into the unique objects that it contains. Additionally, the AI engine knows nothing about the implementation of the object. It treats the object as an image, regardless of the device or platform it comes from. As such, a powerful CV tool is needed and should be supported by an artificial neural network (ANN), a layered structure of algorithms that classify objects. It will train the ANN with many visual objects, resulting in a model that identifies objects it will likely encounter in applications under test (AUT). Thus, when the AI engine is tasked with locating a specific object, it utilizes the model to identify a match in the AUT
In terms of architecture, we implemented the AI engine as a separate module. Rather than restricting it to a specific product, any of our testing products can theoretically use the engine. Our first product that integrated with the AI engine was Micro Focus UFT One, our flagship test automation tool.
Defining OCR-based identification
Our AI engine also leverages OCR to identify text-based objects. These objects may themselves be part of the test, or they could function as a hint to identify the object’s relative location. This capability is useful if an object appears multiple times on a screen. For example, a login screen might have two text boxes, one for the username and one for the password. OCR helps identify which of the edit boxes is which. OCR can also identify a button by its textual caption.
Maturity of test automation based on computer vision
When we rolled out our new computer vision-based testing capabilities, we worked closely with our early adopter customers. We wanted to learn how they were using the technology and to ensure that clients were receiving the greatest value from it. This collaboration helped us discover new use cases and increase the technology’s reach and coverage.
As with any nascent technology, customers reported multiple edge-case scenarios that helped us improve our implementation and refine our model. Over time, we resolved more scenarios and grew the data set we use to train our model. Our algorithms became more capable and reliable. Due to these improvements, these customers now position AI as the preferred test automation mechanism when possible.
Lowering and removing test automation barriers
AI-based test automation reduces the time it takes to build and design tests because objects are identified simply by looking at them. AI algorithms lower skill barriers because they identify most objects and are hidden from the user. Teams can also use the same test without modification on different devices and platforms. They simply procure an appropriate device and run the test on it as-is. And because the algorithm doesn’t rely on an object’s underlying implementation and properties, the test keeps running even if there is a change. If the test’s flow stays the same, the test will continue to run.
The final barrier yet to be removed completely is test execution time. Tests will always take a finite time to run; hence there is a lower limit on the amount of time they take. However, AI-based testing helps teams test earlier and provides robust mechanisms that parallelize and optimize test execution, reducing the wait time for results.
Micro Focus is a Gold Partner at EuroSTAR 2022. It’s our first in-person event in 2 years – and it’s going to be a massive celebration of testing! Learn from 70 testing experts, and connect with your peers at Europe’s best testing event. Get your ticket now.