The Countdown Is On — Secure Your Ticket Now!

EuroSTAR Conference

Europe's Largest Quality Engineering Conference

Test Automation

How to run any number of UI tests on each PR

by

If you are reading this article, likely, you’ve already recognized the value of incorporating UI tests for every pull request in your development process. In short, it’s the single way to be confident that your main branch is ready for release anytime. Releasing excellent and stable versions is crucial in the mobile world, where a user fully manages an app update process, unlike the backend world. If you are a newcomer to the topic of UI Tests for Android, please explore my previous article or Alex Bykov’s talk, where you will find all details and explanations.

While beneficial, running UI tests on each PR introduces significant challenges for the underlying Infrastructure. As a result, almost every team that attempts to implement UI tests for each PR encounters difficulties, often making the same mistakes and investing considerable time and resources in the process. In this article, I will delve into the specific requirements of the Infrastructure needed for running UI tests on PRs and discuss the solutions available in the market. By comprehensively understanding the infrastructure challenges and available explanations, development teams can better navigate the complexities of implementing UI tests for each PR, ultimately saving time and resources.

UI Test Infrastructure

First, let’s define the term “UI Test Infrastructure.” “UI Test Infrastructure” is the thing that allows running the UI tests. On each PR. Any number of tests. From a user’s (Software Engineering and QA Teams) perspective, it looks like I send a command to execute a bunch of tests and receive a report. It’s all. It must be simple for those who use it. But it is complex for those who build and support the solution. So, our final goal is to build this Infrastructure somehow using internal or external solutions and resources.


Okay, now let’s refresh our memory about the entire picture of the UI Test process with some updates that appeared last time.

You see a lot of details in the puzzle. Now, have a look at where “UI Test Infrastructure” is presented.

“UI Test Infrastructure” covers an extensive set of various things. Writing and Backend parts stay on the User side because only the user (developer or SDET) can create tests now. Therefore, writing is out of the scope of this article. An excellent comparison of all writing tools is described in the articles Where to write Android UI tests (Part 1) and Where to write Android UI tests (Part 2) (except Maestro, which appeared after publishing). The backend stuff regarding testing against real or mock networks will be partly touched on later. But I need to mention that Mock Network at scale becomes the responsibility of UI Test Infrastructure too.

Requirements

Before delving into the complexities of building this infrastructure, I recommend to begin with a clear set of requirements and expectations. These will serve as a roadmap to guide you through the intricate process of constructing the infrastructure. Also, it is important to remember that the main users of UI Test Infrastructure are developers, SDETs (Software Development Engineers in Test), and QA. As such, our focus should be optimizing their working experience and ensuring a comfortable and efficient environment for these professionals.

Have a look at the image below.

Now, let’s consider point by point.

Supported platforms

In Android development, there are two primary platforms for creating UI tests:

  1. The native platform, where developers utilize tools exclusively provided by Google, such as Espresso and UI Automator. Solutions like Kaspresso, Barista and etc., are built on top of Espresso and UI Automator.
  2. Appium, an open-source, cross-platform testing framework.

Interface

Next, a fundamental expectation is the ease of integration with existing CI/CD systems through plugins and the ability to utilize the infrastructure from the command line interface (CLI). The plugin and CLI should offer at least the flexibility to filter tests for execution and select the desired devices.

On top of this basic functionality, different verification modes should be supported, i.e. fast runs vs verifying a fix for flakiness. More details about these terms will be provided later.

Reports

At the end of a run, a user expects to see reports that contain at least the following information:

  • the final result: passed or failed
  • number of executed, successful, failed and ignored tests
  • information about failed tests like stack trace, device logs, and video
  • some analytics, like the percentage of failed tests, including retries

Stability

Stability is a comprehensive term encompassing various aspects of UI testing. A test may be unstable (or flaky) due to numerous reasons, such as poorly written tests, an unstable backend or an unreliable network if the test depends on a real backend, feature flags, an improperly set up or unstable device (e.g., a Google service updating or unturned-off animation), framework instability (Espresso, UI Automator, and Appium are known to have their quirks), or internal issues with the test infrastructure caused by factors such as high load or crashes in one of the internal services.

Have a glance at the image below to summarize the possible reasons for failures:

When selecting an infrastructure, we expect full stability or, at the very least, quick recovery that does not impact the overall results and time. The UI Test Infrastructure should cover areas “Where to run”, “Running”, and “Hardware infrastructure”.

Over time, the number of UI tests tends to grow, and with it, the emergence of flaky tests. A flaky test is a test that works most of the time correctly but occasionally fails due to peculiar reasons. Unfortunately, flaky tests are an inescapable reality in UI testing. While it is crucial to investigate the causes, it is not great to block a pull request due to a single occasionally failed UI test. Therefore, as a user of UI test infrastructure, I expect a straightforward and integrated retry mechanism to be available.

Time and Scalability

Test suite execution time is a critical factor for all UI test infrastructures. To better understand this, let’s first examine the elements that influence execution time:

These factors can be divided into two groups: those that depend on the user’s tests and those related to the infrastructure.

Various strategies can be employed to reduce test execution time. One widely-used approach is to focus on specific functionality within a single test by mocking the backend and avoiding repetitive actions such as logging in.

Regarding infrastructure, the test execution algorithm plays a vital role in determining the time required. For instance, consider a test run with a suboptimal batching strategy:

Alternatively, examine a test run with a non-optimal retry policy:

In my recent study, which included over 30 interviews with various development teams, I found that most teams are willing to wait 15 minutes for a UI test run on a pull request. This suggests that even with numerous developers and UI tests, not optimally written tests, concurrent PR runs, or flaky tests, all PRs should be completed within this 15-minute window.

However, it’s a common scenario that PR waiting times can extend from 15 minutes to several hours when the infrastructure is under heavy load, often resulting in rejections due to timeouts. This highlights the importance of optimizing the system to handle such situations efficiently. That’s why I’ve included “Scalability” in the title.

Security

Security protects the test environment, data, and application from unauthorized access, tampering, or malicious activities.

Some critical aspects of security in an Android UI testing infrastructure include:

  • Authentication and authorization: Ensure only authorized users can access the testing environment, data, and resources.
  • Data protection: Safeguard sensitive information, including test data and user credentials, using encryption in transit and at rest. Implement proper data storage and disposal practices to prevent data leaks.
  • Network security: Secure the communication between testing devices, servers, and other infrastructure components.

Cost

I would emphasize the two main aspects:

  • The price model and utilization of paid resources. Generally, there are the following price models: pay for a parallel and pay for a minute. However, choosing and using the appropriate price model wisely is a separate big topic.
  • The second aspect is the internal UI Test Infrastructure algorithms and solutions that allow spending less money running the same test suite. One of the possible optimizations is described above (better batching and handling of flaky tests).

You can find more details about cost price models and specific examples in the article titled “I Want to Run Any Number of Android UI Tests on Each PR: Cost, Part II.”

Support

Last on this list, but certainly not least in terms of importance, is Support. Many teams prioritize not just the service itself but also the quality of support provided. Factors such as prompt responses, a willingness to help, and the ability to save time for the team are highly valued. Additionally, prioritizing features based on client needs and preferences further enhances the overall support experience. Open-sourcing portions of Infrastructure is often highly appreciated by clients, as it fosters trust in the solutions provided and enables them to better understand the underlying mechanics.

Available Cloud Solutions on the market

As an engineer, you have the option to create your own infrastructure that meets all of the above requirements. However, it’s clear that this task can be challenging and complex. Therefore, let’s explore cloud solutions that offer ready-made options.

There are the following Cloud solutions:

Please review the articles below, where I have thoroughly examined each solution based on the aforementioned requirements.

Conclusion

In this article, I examined the concept of UI Test Infrastructure and described the essential criteria for selecting or constructing it. As you might have observed, developing an Infrastructure that fulfills all of the aforementioned requirements and addresses potential issues can be quite complex and challenging. Therefore, I emphasized the existing Cloud Solutions that provide ready-made alternatives.

Author

Prashant Mohan

Evgenii Matsiuk, co-founder at MarathonLabs, co-author of Kaspresso, Android Google Developer Expert.

Testwise are exhibitors in this years’ EuroSTAR Conference EXPO. Join us in Edinburgh 3-6 June 2025.

How Gen AI is Empowering Testers to Work Smarter, Not Harder

by

With software becoming more complex and release cycles getting shorter, traditional testing methods are struggling to keep up. That’s where Generative AI (Gen AI) comes in. Instead of spending hours writing test cases or fixing broken scripts, teams can now use AI-powered tools to create tests, adapt to changes, and catch issues earlier—all with less manual effort.

Gartner predicts more than 80% of enterprises will have used generative AI APIs or deployed generative AI-enabled applications by 2026.

But this isn’t about replacing testers. It’s about making their lives easier, helping them focus on what matters: building faster software. Let’s look at how AI-driven testing is changing the game and what it means for QA teams today.

The Problems with Traditional QA

QA testing has long been a challenge for software development teams. While manual testing provides detailed human insight, it is slow, labor-intensive, and prone to human error. Although automated testing helps speed things up, it comes with its own set of issues, particularly in terms of script maintenance and adaptability.

  • Time-consuming test creation and execution – Writing test cases from scratch is slow and requires significant effort. Running tests across different environments and devices adds further delays.
  • Frequent script failures – Automated test scripts often break when applications undergo minor UI or functionality changes, leading to high maintenance efforts.
  • Lack of scalability – As applications grow in complexity, maintaining comprehensive test coverage becomes difficult. Manual testing struggles to scale, and automated testing requires extensive upkeep.
  • Reactive bug detection – Traditional testing often identifies defects late in the development cycle, leading to costly fixes and delays.
  • High operational costs – The need for large QA teams, expensive testing tools, and ongoing maintenance increases the overall cost of software development.
  • Limited test coverage – Manual and traditional automation approaches often miss edge cases or complex user interactions, increasing the risk of undetected bugs in production.

How Gen AI Transforms QA Testing

The limitations of traditional testing often result in slower development cycles, higher costs, and an increased likelihood of defects reaching end users. To stay competitive, organizations need smarter, more adaptive testing solutions—this is where Gen AI makes a difference.

  • Automated Test Generation
    Gen AI can analyze user stories, requirements, and past test data to automatically create test cases. This reduces the time testers spend writing scripts and ensures comprehensive test coverage. AI-generated tests can even include edge cases that might be overlooked manually.
  • Self-Healing Test Scripts
    One of the biggest pain points in automated testing is script maintenance. When an application’s UI changes, traditional automation scripts break. AI-powered tools detect these changes and automatically update test scripts, minimizing manual intervention.
  • Smarter Defect Detection
    AI doesn’t just run tests—it learns from past failures. By analyzing historical test data, Gen AI can predict where bugs are likely to occur, helping teams focus their efforts on high-risk areas. This means catching issues before they reach production.
  • Natural Language Test Execution
    With AI-based test agents that understand Natural Language Processing (NLP), testers can write test cases in plain English instead of coding them. The AI converts these descriptions into automated test scripts, making test automation more accessible to non-technical team members.
  • Faster Regression Testing
    Automating regression testing is crucial for agile teams. Gen AI enables continuous testing by quickly running thousands of test cases, providing real-time insights and reducing release cycles.

The Future of AI in Software Testing

AI-driven testing is evolving rapidly, and its adoption is expected to grow significantly in the coming years. According to market.us, the global AI in software testing market size is expected to be worth around USD 10.6 Billion by 2033, from USD 1.9 Billion in 2023, growing at a CAGR of 18.70% during the forecast period from 2024 to 2033.

Here’s what we can expect in the future:

  • More advanced AI-powered test agents – AI test bots with enhanced NLP capabilities will allow even non-technical users to create and execute automated tests with minimal effort.
  • AI-driven predictive testing – AI will analyze historical defects, system logs, and code changes to anticipate where bugs are likely to occur, allowing teams to focus their testing efforts more effectively.
  • Increased adoption of self-healing tests – Self-healing scripts can automatically adapt to UI changes, minimizing the need for maintenance efforts and manual intervention..
  • Seamless AI integration with DevOps pipelines – AI-driven testing will become a standard component of CI/CD workflows, accelerating software releases while maintaining high quality.
  • Hyperautomation in QA – Combining AI with robotic process automation (RPA) and machine learning will create highly efficient, fully automated testing ecosystems.
    As AI continues to improve, software testing will become more autonomous, intelligent, and efficient. Testers will shift their focus from repetitive execution to strategic decision-making, ensuring that AI complements human expertise rather than replacing it.

Gen AI Isn’t Here to Replace Testers—It’s Here to Empower Them

One of the biggest concerns surrounding AI-driven testing is the fear of job displacement. However, the reality is quite the opposite. AI is designed to amplify human capabilities, not replace them. Testers play a critical role in quality assurance, and AI is simply a tool to help them work smarter.

Instead of spending hours on repetitive test execution and debugging broken scripts, testers can now focus on exploratory testing, usability evaluation, and strategic test design. AI helps remove bottlenecks, speeds up the testing process, and allows teams to shift their efforts toward more valuable tasks.
Testers are no longer just bug finders; they are quality enablers. AI allows them to do more in less time, ensuring that software is not only functional but also user-friendly, accessible, and secure.

The Next Step in AI-Powered Testing

For teams looking to embrace AI-driven testing, tools like Kane AI offer a game-changing approach. As the world’s first AI-native QA Agent-as-a-Service platform, Kane AI simplifies test generation, automation, and debugging through natural language. By integrating seamlessly with existing workflows, it helps teams create resilient, scalable tests with minimal manual effort—empowering testers to focus on quality rather than maintenance.

The future of testing belongs to teams that adopt AI as a collaborative partner, leveraging its strengths while focusing on delivering high-quality and user-centric software. As Karim Lakhani said: “AI won’t replace you. But someone using AI will.” The key is to adapt, innovate, and lead with AI—because the future of testing isn’t just automated, it’s intelligent.

Author

Mudit Singh

VP of Growth & Product, LambdaTest



LambdaTest are Exhibitors in this years’ EuroSTAR Conference EXPO. Join us in Edinburgh 3-6 June 2025.

Taking Your First Steps with GenAI in Quality Engineering

by

Generative AI (GenAI) is increasingly being recognized for its potential to enhance quality engineering by generating content from existing information to achieve known outcomes. However, determining where to begin can be challenging.

Key Activities for GenAI Implementation

  • Reviewing Requirements/User Stories – Use GenAI to analyze and refine requirements, ensuring clarity and completeness.
  • Generating Test Cases/Scenarios – GenAI can quickly generate diverse scenarios, reducing the manual effort involved.
  • Generating Test Scripts – Generate test scripts, or agile session sheets, that can be used for manual testing.
  • Generating Automation Code – Focus on small functions rather than entire frameworks to incrementally enhance your automation suite.
  • Generating Bug Reports – GenAI can help standardize bug reports, making them more useful for developers and save you time during the execution process.

Choosing the Right Starting Point

When choosing where to start, identify the activity that causes the most pain or time loss and poses the least risk if the outcome isn’t perfect. This strategic choice will free up time to invest in other areas of the lifecycle.

Start Small and Scale Gradually

Remember, the key to successful GenAI implementation is to start small. Master one activity before scaling to others. By doing so, you can gradually build confidence and expertise, ultimately enhancing your quality engineering processes with GenAI.

Which GenAI tool to use?

There are many free or low cost GenAI tools that are available today. Here are some tools you can consider:

  • ChatGPT
  • Gemini
  • Claude
  • Microsoft Copilot

In most cases you can use the free versions of these tools, but be aware that there may be limits on transaction rates, and your data may be used for training. Speaking of which…

Security & Privacy

Data security & privacy is a critical concern for GenAI usage. Some vendors offer paid versions that provide greater data security and privacy features.

Evaluate the data privacy policies associated services to ensure they align with your organizational requirements.

NEVER Put Sensitive or PII Information Into GenAI Tools!

Quick Start guide to Prompt Engineering

So you’ve identified the activity you want to automate, you’ve selected the GenAI tool of preference…Now to write your first prompt!

Use a robust prompt engineering pattern like R.I.S.E. (Role, Input, Steps, Expected Output) to help guide GenAI to produce the desired results. The following is an example prompt for creating high-level test scenarios:

# Role: You are a software tester
 
# Input: I will provide a requirement
 
# Steps: I want you to generate high-level test scenarios. You should ensure that you generate positive and negative test cases, using equivalence partitioning and boundary analysis.
 
# Expected Output: The response should be in a table with the following headings: 
- TS-ID: A unique test scenario identifier starting with "TS-" 
- Description: A description of the test scenario 
- Type: A value of "Positive" or "Negative" that indicates if the test case is a positive or negative scenario. 
- Expected Outcome: The expected outcome from the test scenario

R.I.S.E is just one pattern, so try experimenting with different ones to see what works best for your input & output.

Experiment and Have Fun!

Author

Jarrod Plant

Jarrod Plant is a seasoned professional with 20 years of experience in software testing consulting, providing him with a diverse range of skills and knowledge in various industries, tools, and corporate cultures. He has a technical background, with experience in both Automation & Performance testing, that is balanced by his experience in customer solutions driving a value-focused delivery.

Driven by his passion for the potential of Artificial Intelligence, Jarrod currently serves as the product owner of Planit’s Quality Engineering-centric Generative AI platform. This role provided him with invaluable firsthand experience in building, managing, and testing a generative AI platform. He believes that we are at the precipice of an evolution in software delivery and testing, brought on by the power and accessibility of AI.

Planit are Exhibitors in this years’ EuroSTAR Conference EXPO. Join us in Edinburgh 3-6 June 2025.

The Role of “Digital Tester” in Quality Engineering

by

In today’s world, there is a profound transformation happening in how we compete, create and capture value. With the speed at which technologies like Generative AI and Agentic AI are adopted widely, the whole relationship between humans and machines is getting redefined.

Quality engineering is no exception to this. Adoption of AI into the testing tools, Development and Testing of AI applications and development is on the rise with so many new tools and strategies emerging daily.

Enterprises are competing in the way they are running their entire quality engineering operation. Your competitor is running their QE operations with a team of one-third the size of yours without compromising the scale and accuracy. In fact, they are growing twice as fast as yours. How?

While most enterprises are still deploying likes of ChatGPT for generating content and creating chatbots, very few are fundamentally reimagining the quality engineering with AI. They are deploying the “Digital Tester”. These digital testers are nothing, but the digital teammate or digital colleague implemented using Agentic AI.

While these digital colleagues can drive quality engineering at incredible speed and scale, they also have their own unique characteristics and limitations. Understanding these characteristics not only guides us in terms of what tasks to delegate to these agents but also build a strong relationship that maximize the potential of both human and machines.

While these digital testers are evolving themselves from simple automation tools to the complex autonomous agents, it is very important to select the right digital tester depending on the various factors e.g. your use cases, technical complexity, implementation costs etc. It is very similar to onboarding a new team member and integrating him into your existing team.

Although it looks like a very simple choice, i.e. selecting the complex autonomous digital tester to reduce manual dependency and improve speed, it is a wise decision to opt for a digital tester which supports the entire continuum from automation to autonomy. This will allow flexibility so that we can use the different digital tester skills below as per the testing needs.

  • Predictable and consistent behavior of digital testers with pre-defined rules; No learning and adaptation
  • Digital tester leveraging LLMs for constraint awareness, but behavior is validated against predefined rules
  • Digital tester with reasoning and action; Mult-step workflows are broken down into smaller actionable paths
  • Digital tester’s Reasoning and action combined with RAG for external knowledge sources
  • Integrate Digital tester with multiple tools for leveraging APIs and other software
  • Self-reflecting /analyzing Digital tester using feedback loops
  • Digital tester recalls relevant past experiences, preferences & uses this context for Reasoning
  • Digital testers actively manipulate and control digital/physical environments in real time
  • Digital testers improve themselves over time, learning from interactions, adapting to new environments, evolving

In its simplest terms, the testing needs of any enterprise can be broadly categorized into “What”, “How” and “When” of a software feature and Digital testers with the above skills can help us in all these aspects. AI assisted testing for “What” part of the testing e.g. AI pattern recognition helping the testers to know which parts of the application are likely problematic based on the analysis of the past test cases and historical data. AI powered testing for “How” part of the testing e.g. Self-Healing with AI ensuring the test cases remain valid when changes occur without manual intervention. And AI agents for testing for “When” part of the testing e.g. Self-learning AI with ability to spot unusual behaviour in the application by learning from each test cases they execute or independently exploring the application to discover unexpected issues.

Another most important consideration while selecting the Digital tester is deployment of the digital tester to test AI/ML systems. As AI and ML become more prevalent in our lives, it’s crucial to ensure these systems are thoroughly tested to work as intended.
While selecting your digital tester, make sure that the digital tester can overcome the challenges like being non-Deterministic, Lack of adequate and accurate training data, testing for bias, Interpretability and Sustained Testing and supports the critical aspects of AI systems testing like data curation & validation, algorithm testing, performance and security Testing and regulatory compliance e.g. compliance towards the country’s AI act.

Summing Up

The rise of AI and Generative AI marks one of the most transformative shifts in our lives. Over the past decade, advancements in machine learning, deep learning and neural networks have driven artificial intelligence theoretical concepts into real worlds applications. This evolution has revolutionized quality engineering, where AI became an integral part of the traditional testing platforms and tools. These platforms no longer remained only tools, but they have evolved into a complete Digital Tester which can be part of your Quality Engineering team and work in collaboration with the humans to deliver an exceptional result. As businesses increasingly use AI to construct systems and applications, these Digital Testers are now in turn made to test the AI applications.

The AI testing approaches, procedures and platforms will continue to evolve and improve over the next few years, eventually approaching the maturity and standardization of Digital Testers in the quality engineering landscape.

Authors

Keval Hutheesing, Chief Executive Officer, Cygnet.One

Keval Hutheesing, Chief Executive Officer of Cygnet.One, spearheads the organization’s strategic evolution toward scalable, high-performance technology solutions with quality engineering at its core. His visionary leadership integrates quality throughout the development lifecycle—driving automation, compliance, and operational excellence.
Keval positions quality engineering as the strategic foundation that accelerates business outcomes, ensuring consistent delivery, proactive risk mitigation, and exceptional customer experiences. Through his implementation of a comprehensive quality framework, he propels Cygnet.One’s transformation into a sophisticated platform-driven ecosystem where excellence is intrinsically woven into every aspect of operations.



Shivangi Dubey – AVP & Head of Quality Engineering, Cygnet One

With a rich background steeped in over 15 years of expertise in Quality Engineering and Product Management, Shivangi is a seasoned leader in driving transformative journeys and cost optimization through innovative approaches. She excels in securing new business, executing successful Testing Automation projects, and implementing comprehensive testing strategies.
Renowned for her problem-solving prowess and visionary leadership, she collaborates with customers to expand testing footprints and drive innovation. Experienced in strategic consulting, business development, and process standardization, achieving excellence is her way of life. As the Head of Quality Engineering at Cygnet.One, she brings a stellar track record and an unwavering commitment to excellence.

Cygnet One are Exhibitors in this years’ EuroSTAR Conference EXPO. Join us in Edinburgh 3-6 June 2025.



Business Process Automation for Software Customers – Needs and Challenges

by

Often automated testing is seen as an act by the team that is creating the software. It is a task to ensure quality of the software before it is shipped. However there is an important usage of test automation that can take place AFTER the software is shipped. And it is done BY the CUSTOMER!

The Need:

Customer organizations which buy software don’t just use one particular software. For different aspects of their business, they use different software applications. These applications are integrated together in various workflows, so that they work in tandem to improve employee productivity. Teams in such organizations use these applications on a daily basis and their productivity would falter if any of the software applications fail to function as expected. It is a reality of the current times that each of these applications may be updated 2 to 3 times if not more times a year. And each software application may have its own release cycle/cadence. Any software update may disrupt the employee workflow and cause significant damage in terms of time and money for the company. Hence, organizations need to ensure that their workflows continue to work after any update to these software applications. Such task may be carried out by a business team or a QA team at the customer org.

Can this verification be automated? What are the challenges?

We regularly work with customers in the Automotive industry. These are the challenges faced by them.

Challenges Faced:

1) Software applications in business workflows may be built on different technology stacks. A desktop application may be used for designing automotive parts while a web based PLM application may be used for managing those designs. The interplay between these two applications would be important for the customer teams. Most automation tools do not support automation of multiple technologies.

2) Applications once shipped behave mostly as a black box. They need to be exercised mostly via the Graphical User Interface (GUI). Importantly, we need to mimic the end user’s usage pattern, so GUI becomes significant.

3) Teams that verify such functional continuity will be small and more focused on the business aspects than the automation aspects. They may not have technical know-how (or time) to automate such third party applications using traditional programmatic tools.

So teams look for an automation tool that:

1) Can work across technologies like desktop, web, java, SAP etc.

2) Is easy to use – preferably low or no code

3) Has good support. Since the team may not be very technical, a good support team ensures any edge cases can be handled correctly and quickly

Over the past few years, Sahi Pro has helped a lot of customers achieve such business process automation, especially in the Automotive industry. With the upcoming Sahi Pro v11, automation becomes even more easy because of the no code flowcharts interface. The Flowcharts interface makes visualizing and managing automation very accessible to non-technical testers and business users. Sahi Pro 11 Beta is currently available. Reach out to us to play with Sahi Pro 11 Beta to get a POC done on your automation needs.

Author

Narayan Raman, CEO

Narayan is the CEO and founder of Tyto Software. He is the author of open source Sahi and the architect of Sahi Pro.

Sahi Pro is an exhibitor at EuroSTAR 2024, join us in Stockholm.

Prompt-Driven Test Automation

by

Bridging the Gap Between QA and Automation with AI

In the modern software development landscape, test automation is often a topic of intense debate. Some view it strictly as a segment of Quality Assurance, while others, like myself, believe it intersects both the realms of QA and programming. The Venn diagram I previously shared visualizes this overlap.

Historically, there’s a clear distinction between the competencies required for QA work and those needed for programming:

Skills Required for QA Work:

  • Critical Thinking: The ability to design effective test cases and identify intricate flaws in complex systems
  • Attention to Details: The ability to ensure that minor issues are caught before they escalate into major defects.
  • Domain knowledge: A thorough understanding of technical requirements and business objectives to align QA work effectively.

Skills Required for Programming:

  • Logical Imagination: The capability to deconstruct complex test scenarios into segmented, methodical tasks ripe for efficient automation.
  • Coding: The proficiency to translate intuitive test steps into automated scripts that a machine can execute.
  • Debugging: The systematic approach to isolate issues in test scripts and rectify them to ensure the highest level of reliability.

We’re currently at an AI-driven crossroads, presenting two potential scenarios for the future of QA. One, where AI gradually assumes the roles traditionally filled by QA professionals, and another, where QAs harness the power of AI to elevate and redefine their positions.

This evolution not only concerns the realm of Quality Assurance but also hints at broader implications for the job market as a whole. Will AI technologies become the tools of a select few, centralizing the labor market? Or will they serve as instruments of empowerment, broadening the horizons of high-skill jobs by filling existing skill gaps?

I’m inclined toward the latter perspective. For QA teams to thrive in this evolving ecosystem, they must identify and utilize tools that bolster their strengths, especially in areas where developers have traditionally dominated.

So, what characterizes such a tool? At Loadmill, our exploration of this question has yielded some insights. To navigate this AI-augmented future, QAs require:

  • AI-Driven Test Creation: A mechanism that translates observed user scenarios into robust test cases.
  • AI-Assisted Test Maintenance: An automated system that continually refines tests, using AI to detect discrepancies and implement adjustments.
  • AI-Enabled Test Analysis: A process that deploys AI for sifting through vast amounts of test results, identifying patterns, and highlighting concerns.

When it comes to actualizing AI-driven test creation, there are two predominant methodologies. The code-centric method, exemplified by tools like GitHub Code Pilot, leans heavily on the existing codebase to derive tests. While this method excels in generating unit tests, its scope is inherently limited to the behavior dictated by the current code, making it somewhat narrow-sighted.

Contrarily, Loadmill champions the behavior-centric approach. An AI system that allows QA engineers to capture user interactions or describe them in plain English to create automated test scripts. The AI then undertakes the task of converting this human-friendly narrative into corresponding test code. This integration of AI doesn’t halt here – it extends its efficiencies to areas of test maintenance and result analysis, notably speeding up tasks that historically were time-intensive.

In sum, as the realms of QA and programming converge, opportunities for innovation and progress emerge. AI’s rapid advancements prompt crucial questions about the direction of QA and the broader job market. At Loadmill, we’re committed to ensuring that, in this changing landscape, QAs are not just participants but pioneers. I extend an invitation to all attendees of the upcoming conference: visit our booth in the expo hall. Let’s delve deeper into this conversation and explore how AI can be a game-changer for your QA processes.

For further insights and discussions, please engage with us at the Loadmill booth.

Author


Ido Cohen, founder and CEO of Loadmill

Ido Cohen is the Co-founder and CEO of Loadmill. With over a decade of experience as both a hands-on developer and manager, he’s dedicated to driving productivity and building effective automation tools. Guided by his past experience in coding, he continuously strives to create practical, user-centric solutions. In his free time, Ido enjoys chess, history, and vintage video games.

Loadmill is an Exhibitor at EuroSTAR 2024, join us in Stockholm.

The Deloitte Digital Tester: Lifting the Curtain on AI-powered Automation

by

In today’s technology field it’s almost impossible to discuss the future without mentioning the big impact Artificial Intelligence (AI) will have, specifically Generative AI. The concept behind generative AI is not new, but has seen a breakthrough to a wider audience in recent years, with AI-tools such as OpenAI ChatGPT, DALL-E, Microsoft Copilot becoming available. Generative AI is able to – you guessed it – generate outputs such as text, images, audio and all of their derivatives, which up until recently were the exclusive domain of the human brain.

Generative AI is driving a wave of technological innovation that can be felt far and wide. This is no different for our own field of expertise: Digital Test Management. We’re currently still witnessing the early days of a transformation which will impact most aspects of how we test and validate the software applications we use every day. The importance of ensuring whether an application is working as designed and intended has always been paramount: how can we ensure the trust of people and organizations in products and services if we don’t have the evidence to show for it? Generative AI will play a big role in transforming our ways of testing, allowing people to work more efficiently, automating many tasks and greatly improve test coverage. In the end this will make it possible to find and fix more defects earlier, ensuring a much smoother – and in many cases safer – end-user experience.

To this end, we want to introduce the Deloitte Digital tester, an AI-driven test platform which is capable of supplementing human testers as an automated and independent tester across the entirety of the testing journey!

Introducing the Deloitte Digital Tester

The recent advances have made it possible to pioneer a novel concept: a fulltime, AI-driven ‘digital tester’ that is capable of interpreting requirements and user stories to subsequently create test cases, execute them, log defects and finally report on the result. This is not science-fiction, but already reality. The Deloitte Digital Tester solution was built to operate across all phases of testing and fully integrates with existing testing tools and ecosystems. It can autonomously perform the following tasks:

  • Test Design: generate test cases from requirements and user stories.
  • Test Planning & Scripting: generate automated test cases or convert existing ones.
  • Test Data Creation: generate relevant test data to support test execution.
  • Test Execution: execute test cases and validate the results, all to create defects and support tickets where required.
  • Reporting: show clearly the test execution progress and outcome analysis by generating consolidated reports.

The Deloitte Digital Tester is not intended to replace human testers, but rather supplement them. This brings some key benefits: human testers will be able to focus on defining and validating business focused test strategies, design and architecture. They will be able to execute  more value-adding test cases in parallel to the digital tester, evaluate the overall results and confirm the accuracy. Freeing up human testers also allows for more exploratory-based testing and validation of business interactions in the application. It also keeps testers motivated by automating many repetitive tasks and repeated regression testing. Additionally, the specific skills of the Deloitte Digital Tester allow the solution to do many additional test runs and eliminate blind spots, strongly increasing test coverage while reducing the overall time required to script and run tests.

Benefits of Next-Gen Automation

Our AI-solutions plays a key role in taking Test Automation to the next level. For several years already the concept of Test Automation has firmly rooted itself in the test management sphere: before we had AI, test cases were already being automated to be run as many times as required, and without human input other than creating the automated test cases in the first place and interpreting the results. This allowed for quicker test execution, high test case reusability, improved regression testing and even the creation of large amounts of test data to assist manual testing. With today’s AI-capabilities, the Deloitte Digital Tester is taking the next step: it is able to automate the creation and execution of test cases and evaluate the outcomes. This approach brings many benefits, the main ones we’ll go through here:

  • Continuous Testing becomes a truly integral part of the software development process, rather than testing being handled during specific phases, often post-development. The Deloitte Digital Tester allows us to automate testing activities and ensure quicker execution and more efficient identification of defects. This can be realized through In-sprint automation, where test cases are created while development is still ongoing. These test cases can already expose defects that can subsequently be addressed as early as possible. The sooner a defect is found, the lower its impact and the cheaper the cost to fix it.
  • AI/ML-based Test Data Management utilizes AI and machine learning to optimize the generation of representative test data while at the same time masking sensitive information such as personal data or confidential information. Integrating the Deloitte Digital Tester with the wider testing ecosystem makes the benefits even greater by increasing efficiency across the full landscape.
  • Self-Healing allows automated test cases to automatically update themselves in response to changes in the application’s development. Traditionally automated test scripts required a certain level of human intervention to cover changes in the application being tested. The Deloitte Digital Tester is AI-enabled and scriptless, while at the same time employing machine learning algorithms to dynamically adapt to changes in the application, reducing the need for maintenance. This is particularly relevant in Agile CI/CD environments, where rapid iterations are dependent on efficient regression testing.
  • Increased Test Coverage is realized by AI-driven automated testing. This allows for a scalable and broad-spectrum approach to validate end-to-end (E2E) processes. Additionally, AI algorithms enable us to identify and prioritize test cases based on their potential impact on critical business processes, thereby optimizing test coverage to focus on the areas with the highest risk.
  • Product Validation at Scale is a complex sounding term that signifies how the Deloitte Digital Tester enables organizations to industrialize testing by automating repetitive testing tasks and streamlining the testing process. By standardizing testing practices and creating reusable test assets, organizations can achieve consistency and efficiency in product validation efforts. Additionally, AI capabilities facilitate the analysis of test results and identification of trends or patterns across multiple product lines, enabling continuous improvement of testing processes and product quality.

An Impact that Matters

The above brings us to a key question: what impact does the Deloitte Digital Tester make once an organization chooses to implement it? Especially in case of projects with multiple releases the benefits can be very high when compared to the initial investment. What’s important to note is that the business case behind implementing the Digital Tester will depend on the testing maturity level of the organization. The stronger an organization’s testing capabilities are, the faster the Digital Tester will breakeven and start to provide ongoing benefits and efficiencies compared to traditional automation or manual testing. However, even in case of low maturity levels of adoption, the Deloitte Digital Tester allows an organization to break-even 2-4 months earlier compared to manual testing.

Let’s assume that an organization is running on average 500 test cases per month, and they choose to automate up to 80% of their testing lifecycle efforts.[1] If we are looking at a timeline of 1 year, we can distinguish 2 phases:

  • Ramp-Up Period (4 months in case of high maturity): The Digital Tester requires an initial period during which the solution is trained, and its automation capabilities are being built. If more traditional testing is already going on during this time, this will require more resourcing to maintain these efforts in parallel.
  • Benefit Realization Period (8 months & beyond): Once implemented, efficiencies and automation come into play. This drastically reduces any efforts to automate activities and sets the stage for manual testing to be much more focused and exploratory, for example cross-functional end-to-end-testing, business interaction testing and risk-based deep dives.
    [1] The complexity distribution of test cases in this example is 30% simple, 30% medium, 20% complex, 20% very complex.

In this particular use case, we observe the following quality outcomes by increasing test coverage and next-gen automation:

Future-Proofing your Testing Capabilities

The advent of Generative AI, exemplified by the Deloitte Digital Tester, marks a big step forward in the evolution of software testing and quality assurance. As our world becomes ever more complex, increased consumer scrutiny, regulations and market trends are challenging organizations in many ways to be more fast and agile. In this context, it’s hard to overestimate the increased importance of testing new software applications in the most efficient and thorough way possible. The Deloitte Digital Tester can play a pivotal role in taking your testing capabilities to the next level. This enables you to ensure the smoothest and safest end-user experience for products and services, crucial to maintaining the trust of all stakeholders. It is often said that building a reputation takes years but losing it can happen in a matter of days or even hours.

Rising to meet these challenges, the Deloitte Digital Tester represents a pivotal shift towards a future where AI-driven automation augments human testing capabilities, upgrading the way we approach software testing. As organizations embrace this technological evolution, they not only future-proof their testing processes but also pave the way for innovation and excellence in the way they deliver their products and services.

Contact Details

Thomas Clijsner, Partner, Deloitte Risk Advisory

Tel: +32 479 65 06 96 Email: tclijsner@deloitte.com

Rohit Neil Pereira, Principal, Deloitte Consulting LLP

Tel: +1 916 803 0079 Email: ropereira@deloitte.com

Ramneet Singh, Director Deloitte Risk Advisor

Tel: +32 471 61 89 67 Email: ramnesingh@deloitte.com

Dirk Evrard, Manager Deloitte Risk Advisor

Tel: +32 472 75 92 03 Email: dievrard@deloitte.com

Deloitte is an Exhbitior at EuroSTAR 2024, join us in Stockholm

How Can Copilot Give Your Test Automation A Super Boost?

by

When OpenAI launched ChatGPT a year ago, it made AI more accessible to the broader population and spurred the adoption of AI across various industries. Today, every enterprise is compelled to work with a generative AI strategy and constantly re-evaluate the impact of AI in their industry. The revolutionary generative AI has heralded a new age of applicability in tech, the most recent being Copilot. Copilots, or generative AI-powered assistants, have been steadily rising with generative AI adoption and advancement and are powerful enough to increase productivity and streamline how we work rapidly.  

“AI has undergone a significant transformation in the digital landscape, evolving from a tool for task automation to a collaborative partner. This evolution is driven by advancements in machine learning, natural language processing, and predictive analytics, enabling AI Copilots to understand and predict human needs, thereby providing adaptive and contextually relevant support.”- Forbes.   

An Overview of Copilot

Copilots are advanced collaborative tools based on large language models and generative AI technology. They can integrate smoothly into the existing software infrastructure and manage various activities, from summarizing everyday information to analyzing large datasets. Effective deployment of Copilot helps teams focus on more significant, demanding strategic initiatives. The role of a Copilot is only to augment, not replace, the workforce by helping complete monotonous, labour-intensive, or routine coding activities, thus enriching the overall work experience.  

Copilots leverage AI and align to accelerate code generation based on inputs from user intent and logic. This helps accelerate problem-solving, writing tests, and exploring new APIs (Application Programming Interfaces) without the tedious process of searching for alternate solutions on the web. It is a step ahead and can boost coding efficiencies for developers and testers. It is an intuitive, advanced AI-driven assistant that advances coding abilities by leveraging Machine learning, natural language processing, and advanced analytics.  

Let’s sum this up for you:   

  • Copilot and other programs like it, sometimes called code assistants, assimilate required information and put it right next to the written code.   
  • It tracks the code and comments (or possibly even descriptions written in natural language) in the specific file or files within the same project, assimilates it, and sends this text to the large language model behind Copilot as a prompt.  
  • The Copilot then predicts the expected programmers’ efforts and accordingly suggests the code. 

Transitioning into test automation with Copilot.   

Large language models have gained popularity in various domains, including test automation, and help focus on the quality and reliability of software products. With increasing complexity among applications deployed, the demand for efficient and effective test case creation methods has consequently gained prominence. Generative AI tools, such as OpenAI’s ChatGPT and GitHub’s Copilot, have emerged as powerful and promising resources to accelerate test automation efforts.  

 While enterprise applications can be customized to an organization’s unique needs and processes, setting up Copilot may get tricky with constant release updates, new patches, and configuration changes.   

These custom tweaks still need to function post updates and getting them tested every time can be complicated. Test automation can eliminate the hassles of validation chaos and save time with minimum effort.   

An AI Copilot has its:  

Primary Use Case: Can execute actions.  Core technology strength: Real-time communication with APIs + SQL databases + Vectorized database.  

AI offers remarkable opportunities to streamline and enhance software testing, from script creation to execution. With an increasing rate of technical complexity change and reduced delivery cycle time, AI-powered solutions are stepping up to meet demands for continuous testing.  

QA engineers today have the challenge of updating their test automation suite before each release cycle. This challenge arises from smaller cycles coupled with faster changes- imagine fuelling a car when it is on track. With AI algorithms and automated test script generation, it is possible to keep pace with development and predict potential problem areas before they arise.  

 With AI, QA practices help manage fast-paced development cycles and improve overall quality assurance strategies covering UI (User Interface) Testing and API (Application Programming Interfaces) Testing. With its inherent ability for pattern recognition and predictive analysis powered by machine learning, AI can significantly enhance test coverage, allowing QA teams to focus more on complex tasks and less on mundane, repetitive activities. 

Can Copilot replace developers and automation testers?  

Just visualize a qualified Copilot in an airplane, seated and working in complete tandem with the pilot in charge of the flight. AI Copilots work alongside the user; the developer or the tester takes over basic tasks, handles basic questions, and makes suggestions.  

 Copilots herald a whole new era of testing in day-to-day development, thanks to the advent of AI and ML. Testers and developers can focus on real problems and avoid haggling with the text editor or googling for code completion. Despite its advantages, Copilot could churn out ineffective or invalid code with subtle bugs since it draws from the context it receives as input. AI must still catch all complex nuances a tester or developer can grasp and parse effectively. However, this can rapidly change once such services advance their capabilities with the help of data gathered from skilled users.  

Copilot could have its limitations, but it represents a significant step ahead in harnessing the power of AI in software testing. In an evolving technological environment, Copilot and similar AI tools can become invaluable assets in the testing arsenal, provided the right balance between Automation and human intervention occurs. Copilot is another tool that highlights what can be achieved, but realizing the knowledge and potential lies in the skill of resources.   

 A recent Microsoft report aptly points out that AI cannot fix challenges at work magically; employees still must be adept at critical thinking, creativity, and analytical judgment.   

 So, how can this Copilot super-power be optimized in development and testing areas? 

An all-around automated testing approach is step one.  

Innovative AI solutions can automate testing at scale and help with high-quality software outcomes. AI-powered platforms like Copilot analyze comments and prompts, create test cases, and go through the testing process in a self-healing environment. This does away with flaky testing, enhances test coverage, and improves productivity.   

ACCELQ Copilot 360 is the industry’s first full-cycle, holistic AI-powered test automation solution that goes beyond code generation.  

Features:    

Zero Boilerplate, Perfect Output: AI-Powered Logic Development   

  • No more wordplay hassles: AI-driven development lets one use regular English for scripting. 
  • Complex passages with inter-dependent information   
  • Tolerant to user errors with auto-reconciliation with application state   
  • Real-time validation of developed logic   

Code generation with Design-First: AI for Design and Sustainability 

  • Modularity recommendations to achieve well-designed assets.   
  • AI-assisted parameterization for maximizing re-use.   
  • Standardized naming convention for asset consistency.  

Uninterrupted Execution for Unmatched Reliability: AI for Execution Resilience   

  • Execution reliability is made possible by a closed-loop feedback engine   
  • Freedom from application DOM dependence   
  • Smooth, unsupervised executions in complex environments   

    Proactive Change Management for Seamless Automation: AI at Work   

  • Multi-modal LLM capabilities for change analysis   
  • Proactive suggestions for change adaptation   
  • Minimized disruption to test automation.   

    Asset Quality Evaluation Against Best Practices: AI-Driven Insights  

  • Analyze existing test suites for optimization opportunities.   
  • Identify duplication and redundancy.   
  • Score the project against a best practice baseline.   

If you are eager to experience the power of ACCELQ Copilot 360, contact us and take the first step towards revolutionizing your test automation. 

Author


Geosley Andrades, Sr. Director, Product Evangelist at ACCELQ

Geosley is a Test Automation Evangelist and Community builder at ACCELQ. Being passionate about continuous learning, Geosley helps ACCELQ with innovative solutions to transform test automation to be simpler, more reliable, and sustainable for the real world. 

ACCELQ is an EXPO Exhibitor at EuroSTAR 2024, join us in Stockholm.