EuroSTAR Conference

Shift Left with Smart Test Selection

March 27, 2026 by Lauren Payne

Software testers face a constant dilemma: software grows, but release cycles shorten, leaving less time for testing. »Shift Left« promises a solution, aiming for earlier, more frequent testing. But extensive end-to-end test suites can take hours or weeks, making this infeasible. Luckily, Test Selection offers a proven solution.

The key idea is to run only a small, highly effective subset of your full test suite. We design this subset to require only a fraction of the full suite’s runtime, while concentrating a significant amount of bug-finding power.

While such a small subset may not reveal all bugs, it finds most of them much faster. And when we run this smaller subset frequently, in addition to less frequent full test runs, we get rapid feedback on the majority of bugs without missing crucial issues. This even enables running tests on feature or developer branches, where end-to-end tests were previously uneconomical, further accelerating development.

We tested many Test Selection approaches to find those that deliver good results and are easily applicable at scale, even in legacy industry projects. Our Software Quality Platform Teamscale implements the best approaches to help our customers regain fast feedback from their tests.

Quality Gates vs. Change-Based Testing

In working closely with dozens of teams, we found two main use cases for Test Selection.

Quality Gates: We select a fixed set of tests to run repeatedly over a longer period of time (weeks, months), allowing us to protect valuable resources, such as:

Expensive test runs. To avoid wasting costly test runs on high-level bugs that may mask many other bugs, only software versions passing our quality gate proceed to expensive testing, thereby reducing execution costs.
Integration branches. To ensure stable shared branches, only changes passing the quality gate are merged, preventing buggy code from hindering team productivity.

Change-based Testing: We select a different set of tests for every test run, specifically for the code changes under test. This is more precise, especially for smaller changes. However, it requires a more complex setup to dynamically incorporate change information.

We can even combine both strategies, e.g., by using quality gates before full test runs and change-based testing on feature branches and pull requests for thorough, rapid feedback and comprehensive quality assurance.

Build a Quality Gate with AI Vector Embeddings

End-to-end test suites, often grown through years of copy-and-paste, typically contain numerous similar tests. To identify high-level bugs, we run only one test from each cluster of such similar tests, i.e., we run the most dissimilar tests. This allows us to cover significant software functionality with fewer tests, increasing our chances of finding bugs throughout the software system.

Technically, we represent tests as numerical vectors in a multi-dimensional space, leveraging Large Language Models (LLMs) to generate embedding vectors that capture the semantic meaning of your tests. This works directly on the source code of your automated tests from your version control system (VCS) or on manual test instructions from your Application Lifecycle Management (ALM) system. We then select the test cases that are “furthest apart” in the vector space, to ensure a diverse and non-redundant subset.

Our experiments on open-source and industry systems show that this approach enables teams to find 90% of bugs in just 13% of the full test suite’s runtime. This enables you to turn a long-running and expensive test suite into a quality gate that you can quickly run before every merge.

Test Change-based with The Power of a Search Engine

We can improve further by selecting tests specifically for the changes under test, which we extract from your VCS. From these changes, we generate a search query, by extracting relevant terms (e.g., »search«, »login«, »user«) from modified functionalities. Leveraging existing search engine research, we index your tests (automated test source code or manual test instructions) into a searchable database. Then we query this database, to compute a ranked list of tests, with the most relevant ones at the top, much like Google finds you the most relevant websites. Given your test budget, e.g., 15 minutes, we then run the most relevant tests that fit into this time.

Our experiments on open-source and industry systems show that this approach enables teams to find 90% of bugs in just 4% of the full test suite’s runtime. This may transform even a once-a-week test run into fast, per-pull-request checks, ensuring thorough testing very early on. The only downside, compared to a quality gate, is the increased setup effort, as your test runner or manual testers need to query for the relevant tests at the start of each test run.

Summary

The imperative to »Shift Left« requires smarter testing strategies. Test Selection enables you to run smaller, yet highly effective, subsets of your tests more frequently, thereby accelerating feedback on most bugs. Teamscale offers Test Selection to build quality gates (to protect resources like shared branches or costly test runs) as well as to test specific changes (for rapid feedback on pull requests or feature branches). Both approaches, or a combination, enable »Shift Left« even for extensive end-to-end test suites, ensuring faster feedback and higher software quality without increasing testing costs.

Would you like to dive in deeper? Reach out or visit our Teamscale booth at EuroSTAR 2026!

Author

Johannes Veihelmann

Johannes Veihelmann is a consultant of CQSE GmbH for software quality. He obtained his Bachelor of Science degree in Bioinformatics and is part of the Test Intelligence Team. There he helps customers daily to successfully set up Test Selection analyses.

CQSE are a participating in EuroSTAR Conference 2026 as an exhibitor

How Artificial Intelligence is Rerouting Quality Assurance

March 25, 2026 by Lauren Payne

AI QA is reshaping software testing by bringing intelligence into every stage of the development lifecycle. By combining AI and machine learning, QA teams are moving from brittle automation to adaptive, predictive strategies that catch bugs earlier, reduce test maintenance, and speed up releases.

This post breaks down how Artificial Intelligence (AI) in Quality Assurance (QA) is transforming software testing from smarter test case generation and faster defect prediction to continuous optimization. You’ll also see how teams can start applying AI in practical ways.

What is artificial intelligence in quality assurance?

AI QA refers to the integration of AI and machine learning (ML) into quality assurance workflows. Practically speaking, it’s about using AI to take repetitive tasks off the QA team’s plate, giving them more time to focus on activities that require human insight, like exploratory testing or evaluating edge-case behavior.

A few of the jobs AI QA can perform include:

Generating test cases based on user behavior, system logs, requirement, or recent code changes.

Predicting failure points by analyzing historical defect data, commit patterns, and code complexity.

Triaging bugs automatically using NLP (Natural Language Processing) to group related issues, flag duplicates, and suggest likely root causes.

Prioritizing test execution based on risk scores, code velocity, and business-critical areas to reduce unnecessary test cycles.

Maintaining and evolving test suites by identifying outdated tests and generating new ones in response to product changes.

Why do QA teams need AI now?

Modern QA teams aren’t lacking tools. They’re lacking time, visibility, and actionable insights. Release cycles are getting shorter, systems are more complex, and user expectations continue to rise.

Here’s how that pressure shows up in practice:

Tests are running, but the value is unclear.

Automation is fragile and fixing brittle test scripts often takes time away from coverage.

Bugs still make it to production when testing isn’t aligned with real-world risk.

QA becomes a bottleneck when teams are expected to sign off without enough time, data, or confidence.

Leadership can’t measure impact without clear metrics.

AI QA helps solve these problems by enabling teams to work smarter, not harder. Instead of adding more scripts or expanding headcount, AI reduces waste and helps teams zero in on what really matters. The result is faster feedback and higher-quality releases.

How AI is helping teams improve efficiency

Here are five ways AI QA is helping teams improve efficiency and focus on what matters most:

Smarter test generation: AI tools analyze usage patterns, code changes, and defect logs to automatically generate test cases, saving time and improving test coverage.

With TestRail AI, you can generate first-draft test cases from your requirements text, then review and refine them before adding them to your suite.

Faster defect prediction: By modeling factors like code churn, commit frequency, and historical defect density, AI highlights high-risk areas before issues reach staging or production.

Intelligent bug triage: Using NLP, AI groups related bugs, flags duplicates, and suggests likely owners, helping teams resolve issues faster and reduce backlog noise.

Risk-based test prioritization: Rather than running every test on every build, AI assigns risk scores and ranks test cases based on business impact, recent changes, and failure likelihood.

Continuous test suite maintenance: AI flags outdated/redundant tests to reduce false positives and maintenance overhead.

The strategic edge: QA leaders are tracking and testing

QA teams are taking a more strategic approach. They’re seeking better visibility into what’s working, what’s wasting time, and where risk is hiding. AI is helping leaders track metrics like:

Test debt velocity: How quickly are tests becoming outdated, and how does that affect confidence in test results?

Risk-based test ROI: Which tests are consistently catching critical bugs—and which ones are just noise?

AI vs. manual performance: How do AI-generated tests compare to manual ones in terms of defect yield and maintenance cost?

Suite stability trends: Where is test flakiness increasing, and what are the patterns behind it?

How can my team start using AI in QA?

Start small with efficiency. The most effective teams begin by identifying where AI can have the greatest immediate impact and then build from there.

Map your friction. Where are you losing speed or confidence today?

Pick one high-leverage use case. Flakiness detection and test generation are great entry points. One simple starting point is using TestRail AI to draft test cases from requirements, then standardizing them with a human review step.

Choose transparent tools. Make sure your AI doesn’t introduce black-box risk.

Connect everything to TestRail. Use it as your system of record to track, trace, and manage your evolving strategy.

How TestRail helps teams create an AI QA strategy

AI QA tools can generate tests, flag risks, and optimize execution, but they work best within a structured system. TestRail brings those insights all together, helping teams turn AI QA into a repeatable strategy that scales across teams and release cycles.

Here’s how TestRail works for AI QA:

Track and generate tests in context: Use TestRail AI to draft test cases from requirements text, then manage AI-assisted and manual tests together with full visibility into history, execution, and ownership.
Visualize test coverage by risk: Filter by release, component, or risk category to see gaps and trends.
Centralize automated results: Connect TestRail to your automation and CI/CD pipeline to centralize reporting across automated test runs.
Maintain end-to-end traceability: Link test execution to requirements, defects, and user stories for complete accountability.
Report with clarity: Use dashboards and custom reports to surface performance trends, identify bottlenecks, and share QA impact across teams.

TestRail is built so that speed is measurable. Even as complexity grows and scales, your team stays in control.

Integrate a streamlined workflow with TestRail

Quality assurance measures real-world risk and complexity. Platforms like TestRail let you leverage AI QA without losing visibility, giving you tighter feedback loops and more confident releases. See it first hand, start your free 30-day trial today.

Author

Patrícia Duarte Mateus

With more than a decade of experience in Software QA and expertise in several business areas, Patrícia Duarte Mateus has a QA mindset built by the different roles she has played—including tester, test manager, test analyst, and QA engineer. She’s Portuguese, living in Portugal, and is currently a Solution Architect and QA Advocate for TestRail. Patrícia is also a speaker, mentor and founder of a project whose objective is to demystify and educate on Software QA with a focus on Portuguese-speaking people, called “A QA Portuguesa”. Her areas of interest beyond QA include deepening her knowledge of psychology, tech, management, teaching/mentoring, health, and entrepreneurship. Books, podcasts, Ted Talks and YouTube are always on Patrícia’s to-do list to ensure a good day!

TestRail are a participating in EuroSTAR Conference 2026 as a Gold Sponsor. Join us at EuroSTAR Conference EXPO in Oslo 15-18 June 2026.

Agentic AI for Production-Grade, Domain-Specific Test Automation

March 23, 2026 by Lauren Payne

For the past few years, AI in testing has been dominated by impressive demos. A model generates a test script from a user story. A chatbot repairs a broken locator. A prompt magically produces automation code.

But when enterprise teams attempt to deploy these capabilities into real CI/CD pipelines, they quickly discover a gap between possibility and practicality.

Large Language Models (LLMs) are powerful, but they introduce serious challenges in production testing environments. They are inherently probabilistic; identical prompts can yield different results, undermining regression reliability. They can hallucinate syntactically correct but logically flawed automation steps. As test flows grow longer and more complex, models may lose context. Inference latency and cost can escalate. Model drift can alter behavior over time, requiring continuous revalidation.

Small Language Models (SLMs), by contrast, are fast, stable, and cost-efficient. They perform exceptionally well in structured, domain-specific tasks. However, they lack the deep reasoning power required for complex, multi-step intent interpretation.

So the real question is not “LLM or SLM?”
It is: How do we combine both intelligently to build reliable AI-powered automation?

The Hybrid Architecture: Precision Meets Reasoning

Production-grade AI testing requires architectural discipline.

The most effective approach is a hybrid model strategy:

Custom SLMs trained on domain-specific automation assets—keywords, UI definitions, reusable components, and workflow logic.

LLMs are reserved for advanced reasoning tasks, such as decomposing high-level test intent into structured automation steps.

This separation of responsibilities delivers determinism and efficiency for routine automation while preserving flexibility for complex reasoning. The result is faster inference, lower operational cost, improved stability, and reduced hallucination.

Rather than relying on a single general-purpose model, a hybrid architecture assigns the right model to the right task.

This architectural pattern can be applied to modern automation frameworks in general. At DWS, we have successfully implemented this hybrid model within TestArchitect (TA), demonstrating how domain-trained SLMs and reasoning-driven LLMs can operate together inside a structured automation ecosystem.

TestArchitect supports a wide spectrum of platforms and technologies, including SAP, Salesforce, mobile native, mobile web, mobile hybrid, browsers, and desktop applications. Teams can create fully no-code automated tests in plain English, allowing business analysts, domain experts, and testers of all skill levels to contribute without coding barriers. This approach accelerates in-sprint automation, helping teams keep pace with rapid release cycles while ensuring high-quality testing across diverse applications.

From Single-Model Scripts to Agentic AI

Even hybrid models alone are not enough. True production-grade automation is not generated by a single model guessing a script; it is orchestrated.

Agentic AI introduces a multi-agent system where specialized AI components collaborate across the testing lifecycle:

Interpreting business-level test intent
Mapping intent to reusable automation actions
Generating structured UI interaction steps
Validating and refining codeless automation
Executing across platforms
Performing intelligent self-healing
Classifying root causes of failures
Predicting impact and suggesting remediation

Instead of relying on a single probabilistic output, the system validates itself through structured collaboration. Each agent has a defined responsibility, reducing instability and improving reliability.

Within CI/CD pipelines, intelligent test selection based on code impact analysis ensures that only relevant test suites are executed, reducing cycle time while preserving coverage.

This marks a shift from AI as a script generator to AI as an orchestrated testing system.

Within TestArchitect, this agentic hybrid model powers end-to-end intelligent automation, but the architectural principles are transferable to other enterprise automation environments.

Data Engineering and Model Validation

The real differentiator in AI-powered testing is not model size. It is data quality and governance.

For domain-intensive industries such as finance, healthcare, and energy, AI must align precisely with industrial workflows. This requires:

Structured automation datasets
Clearly defined evaluation metrics
Domain-specific fine-tuning
Human-in-the-loop validation

Domain experts play a critical role in guiding quality standards and ensuring AI-generated outputs meet enterprise expectations.

To improve precision and consistency, Retrieval-Augmented Generation (RAG) is applied using structured project-specific automation data. Instead of relying solely on model weights, the system dynamically retrieves relevant context, existing action libraries, interface definitions, and project artifacts before generating tests.

This approach reduces hallucination, preserves framework alignment, and improves maintainability.

In TestArchitect implementations, this structured RAG strategy has enabled reliable AI-assisted test generation aligned with domain rules and enterprise governance standards.

Deployment, Cost, and Sustainability

Enterprise adoption requires more than innovation; it requires sustainability.

Custom SLMs can be hosted in private cloud or on-premises environments, ensuring that sensitive QA artifacts, requirements, test cases, and defect logs remain within organizational boundaries. Strategic collaboration with cloud-hosted LLMs minimizes unnecessary inference costs.

Because SLMs are lightweight and CPU-friendly, organizations avoid GPU-heavy infrastructure and excessive cloud spending. Millisecond-level responses enable seamless integration into Agile and DevOps workflows.

The hybrid agentic approach ensures that AI-driven automation remains:

Secure
Cost-efficient
Predictable
Scalable

Without architectural discipline, AI in testing becomes unstable and expensive. With the right design, it becomes a long-term competitive advantage.

Engineering the Future of AI-Driven Testing

AI will not replace automation frameworks. It will augment them.

The organizations that succeed will not be those experimenting with prompts, but those engineering intelligent systems, combining hybrid models, agent orchestration, structured data pipelines, and rigorous validation.

This hybrid, agentic AI architecture can be applied across automation ecosystems. Its successful implementation within TestArchitect demonstrates that production-grade, domain-specific, cost-controlled AI testing is not theoretical—it is achievable today.

The future of testing belongs to teams that treat AI as an engineering discipline.

Get Started with TestArchitect AI Today

Author

Tuan Truong – Head of Test Architect Product Development

He leads the design and evolution of enterprise-scale test automation solutions, helping global organizations modernize quality engineering practices and deliver large-scale systems with confidence. With over 20 years of experience in software testing, automation architecture, and product engineering, Tuan specializes in integrating AI into practical testing workflows. His current focus is on designing production-grade AI systems that combine Custom SLMs, LLMs, and Agentic AI to create scalable, cost-efficient automation solutions. He works closely with enterprise teams to transform emerging AI capabilities into reliable, real-world testing systems.

DWS are a participating in EuroSTAR Conference 2026 as a Gold Sponsor

How We Learned to Test Our RAG (and Accidentally Tested Our Content)

March 20, 2026 by Lauren Payne

In the last four years as a developer advocate at Qase, I’ve written more blog posts and LinkedIn articles than I can count. I’ve also reviewed hundreds of pieces from colleagues, checking whether the arguments hold, the examples work, and the claims are backed by evidence.

I got better at it over time. My earliest articles are nowhere close to my latest ones, both in knowledge and in writing style. Naturally, I tried using LLMs along the way. I quickly found out that LLMs can’t be trusted to write on my behalf, and they certainly can’t be trusted to fact-check. They hallucinate confidently, they miss nuance, they produce text that sounds right but often isn’t.

I did find one pattern that works: using LLMs as critics. I write, then I ask the LLM to poke holes in what I wrote. The diversity of their criticism is genuinely helpful. Even when they hallucinate in their feedback, it doesn’t matter, I still have to go through every point and decide what to keep. The hallucinations become noise I can filter, not errors that leak into my content.

This was my comfortable setup for a while. Then my friend Anupam Krishnamurthy showed me something that changed how I think about content quality entirely.

The Spark

Anupam and I co-own BeyondQuality, an open research community for software quality topics. He presented his research on evaluating RAG systems using a framework called Ragas.

For those unfamiliar: RAG (Retrieval Augmented Generation) is a way to make LLMs answer questions from your own documents. You chunk your content, store it in a vector database, and when someone asks a question, the system retrieves the relevant chunks and feeds them to the LLM to generate an answer. Ragas is a framework that evaluates how well this works: does the answer stick to the retrieved content? Did the system retrieve the right content in the first place?

Anupam built a RAG on the 37signals employee handbook and used Ragas to evaluate it. This was eye-opening. I started thinking about LLMs and content quality in a completely new way.

From Curiosity to Production

My first experiment was pure fun: I built a chat interface to a Deming book I own. It worked, but there was no real business case. I know what Deming writes; I’d rather just re-read the book.

The second idea had a real business case. At Qase, we have over a hundred blog posts, a Help Centre, and a customer support knowledge base. Every new piece of content operates on top of everything published before. If a new article contradicts something from six months ago, readers notice. It confuses people and erodes trust.

This is a classical regression problem, just in content, not in code. A company with hundreds of published pieces has the same problem as a codebase with no tests: changes go out unchecked against existing behavior. Before this, the only check was someone reading a draft and trying to remember if it contradicts anything. With a hundred articles, that’s not realistic. With RAG, I could retrieve just the relevant pieces from the existing corpus and check the new draft against them automatically. Ragas then evaluates how well the retrieval and checking actually work. The feedback loop went from “hope someone remembers” to a 25-minute automated run.

What Happened When We Tried It

The first obstacle was building the evaluation itself. To test whether the system answers correctly, you need ground truths: the key claims and ideas from your content. I tried using several LLMs to extract these from our 106 blog posts. They all gave inconsistent results. The LLMs couldn’t agree on what the articles were actually saying. I ended up reading every article myself and writing down the key ideas manually. There is no shortcut here yet.

Then I had to write evaluation questions. This turned out to be the same problem as writing tests after the code is already written: since you already know the system, you’re inclined to write tests that just confirm what’s there. My first questions were like that. They assumed knowledge of the content and led toward the answer. It took three iterations to learn to write questions the way a real person would ask them: short, simple, with genuine uncertainty about which way the answer goes.

Once I had 240 questions and the evaluation was running, the results told a clear story. On broad, open-ended questions, the LLM stopped relying on the retrieved content and started answering from its own general knowledge. It sounded confident and correct, but it was no longer grounded in what we actually wrote. Ragas caught this. Without measurement, we would never have noticed.

The whole evaluation for 240 questions costs $0.60 and takes 25 minutes — I’ve seen classical automated tests’ suites running for longer!

And then the biggest surprise. For some questions, the system couldn’t find the right article to answer from. Not because the retrieval was broken, but because we simply hadn’t written about those topics well enough. We set out to test the RAG. We ended up finding holes in our own content.

Where This Is Going

This research started with Anupam. Without his curiosity and his work at BeyondQuality, I would not have explored this direction at all.

Today we have regression testing for our content: the blog, the Help Centre, the CS knowledge base. New drafts get checked against everything we’ve already published. What started as a contradiction checker is now also helping us find gaps in our existing content we didn’t know were there.

If you want to see how RAG evaluation works hands-on, come to Anupam’s live demo at EuroStar in Oslo this June, 2026. He will walk through building and evaluating a RAG system step by step. And if the intersection of software quality and AI interests you, follow our work at BeyondQuality, where all research is published openly.

I’ll be at the Qase booth throughout the conference. If any of this sparked your curiosity, come say hi.

Author

Vitaly Sharovatov

As a quality enthusiast, I believe that people should take pride in their work and companies should aim to produce high-quality products. I have spent the last 24 years in IT, focusing on engineering, quality assurance and mentorship. I am also a huge animal lover and have saved and raised more than 50 cats and dogs.

QASE are a participating in EuroSTAR Conference 2026 as a Gold Sponsor

From faster rituals to deeper thinking: how AI is changing software testing

March 4, 2026 by Lauren Payne

With agentic AI augmenting testers, I see testers starting to move up the value chain: I see them spending more time thinking about what should be tested and why, rather than just how. I see them reasoning more about systems, users, and risks, instead of pouring time into automating repetitive checks.

In short, with agentic AI, the effort is shifting away from the how toward the what and the why—the higher-value thinking work. Mapped onto a simple model, that shift looks something like this:

Below are a few of my observations on the biggest “what’s” and “whys” I’m seeing in this shift—and why I’m encouraged and genuinely optimistic about it.

AI is reducing the cost of automation in testing

It’s getting increasingly easier to automate the mechanical parts of testing—whether that means automating manual repetitive checks, generating test data, or finding patterns in test results. And that’s the aspect I like most about AI in testing. Not because the machinery of automation gets cheaper, but because something more important happens: testers finally get the time and space to think.

They can step back, look at the whole system, and spend their cognitive energy where it truly counts: finding software problems that matter.

Scalable automation reveals the value of skilled testing

AI doesn’t just reduce the cost of generating automation. It exposes the real value of professional human testing: testing that consistently finds rare, hidden, and subtle problems that matter. And as AI automates more of the repetitive and dreary work, testers are shifting upward from “How do we automate this?” to “What risks matter most to which stakeholders?”.

The bottleneck isn’t automation, it’s thinking

And this shift—this reallocation of attention—is healthy, because the bottleneck has never been automation, it’s thinking. What limits how fast we can ship high-quality software consistently is the time required for deep exploration, experimentation, and investigation. That’s what surfaces the risks that matter.

Teams rarely fail because they can’t write enough automation. They fail because they don’t (or can’t) ask the right questions soon enough.

Do we understand the system?
Do we understand the users?
Do we understand which risks matter most to which stakeholders?

If we don’t ask those questions, no amount of automation will save us.

Testing is an investigation, not a production line

Testing is not a factory process. It’s an investigation into quality, risk, and information we don’t yet have. The challenge in testing is framing risks, designing experiments, questioning assumptions, and telling a useful story about the quality of the software we are testing.

And we do all of that with one goal in mind: to enable other people (like developers and managers) to make better-informed decisions faster (whether that’s fixing something or shipping it). That’s the heart of testing. AI supports it, but it doesn’t replace it.

AI is an amplifier, not a replacement

This is why I see AI as a capability amplifier, not a replacement for software testers. It can suggest relevant tests, draft automation, generate data, and summarize logs. AI produces useful artifacts and insights, but people produce judgment.

AI raises the ceiling of what testers can do; it doesn’t eliminate the need for human sense-making. Again, AI is an amplifier of everything; it increases ingenuity…but also risks. When AI becomes more autonomous, it becomes easier to mistake activity for insight. If we want to get the most out of this technology, we also need to understand the conditions under which it can mislead us. Not to be cautious, but to help us steer AI in the right direction.

When quantity masquerades as confidence

One of the first ways AI can misguide us is by creating an illusion of test coverage. When test ideas become easy to generate, it’s tempting to equate test volume with test quality. But professionals know that a mountain of AI-generated checks can still miss the real risk: coverage numbers can look impressive, dashboards can glow green, yet critical questions may remain unanswered.

Sure, we can ground AI in our context. But most of our knowledge is implicit—the kind that sits in people’s heads, not in documents. AI can’t magically access that. So, don’t let AI dictate your testing. Treat every AI-generated test idea as a prompt for inquiry: “What does this miss? What does this assume? What does this not know?”. Guidance, not obedience, is the tester’s job.

Faster automation can still be the wrong automation

Scalable automation encourages the automation of everything, including things that don’t matter. Professional testers know this well. As Dorothy Graham says: “If you automate chaos, all you get is faster chaos.” Professional testers don’t create automated checks for their own sake. They don’t measure value by volume. They know that a small set of risk-revealing checks is far more valuable than a large set of automated checks that tells you little or nothing about real risk. They ask: “What information does this check give us?” or “Does this tell us something meaningful about risk?”. They focus on automating what matters, not just what’s easy.

The hidden risk: AI can deskill software testers

Another subtle trap is the quiet erosion of skill. Professional testers understand that AI can de-skill them if they stop practicing the hard part. They know if they mostly prompt AI to spit out test ideas, they may not build the muscles for modeling unfamiliar systems, recognizing ambiguity and bias in requirements, or designing experiments under uncertainty.

And when those muscles fade, so does the ability to judge AI’s suggestions. The twist is that AI frees professional testers for higher-value work, while it can also mislead amateur testers—less experienced testers or those testing from other roles (myself included)—into trusting AI’s output too literally. Without solid testing skills, it’s easy to mistake fluent suggestions for sound ones.

Overall, AI amplifies what testers bring to the table. It multiplies strengths and gaps. Mixed teams of professional and amateur testers that acknowledge these asymmetries will thrive.

The choice: better thinking or empty rituals?

This brings us to the decision testing teams must make. AI creates a new choice point: do we use the time it frees up to invest in better thinking (stronger modeling, sharper questions, better risk analysis) or do we spend it on more rituals?

By ritual I mean busywork disguised as testing: generating piles of automated checks because it’s easy, following AI-generated test plans rigidly even when new risks emerge, or filling dashboards with metrics that don’t reflect real quality. The future of testing depends on this choice.

Closing the loop: what makes me optimistic

So let me return to where I started: AI is changing testing, not by replacing testers but by revealing what they do. And when I look at the teams I work with, I see something encouraging: a few years back, in our backlog refinement meetings, I’d see testers’ first question was always “How do we automate this?” instead of “What should we test and why?”.

But now, I’m noticing something different. As it’s gotten easier to automate, the conversation has shifted. More testers are asking those what and why questions because they have the breathing room to do it.

More conversations are about risk rather than scripts. More energy is going into making sense of the work than making checks. More testers are choosing deeper thinking over empty rituals.

Yes, AI has its risks (illusion of coverage, wrong automation, deskilling). But I see software testers responding thoughtfully. They are using AI to deepen their judgment, not to avoid it.

The net effect? Testing is getting better, not because AI is magical, but because it frees testers to do more of the work that matters: work that doesn’t just produce more automation but surfaces more truth about the quality of our software; work that doesn’t just go faster but goes deeper.

And that makes me optimistic about the future of our craft.

Author

Ingo Philipp, UiPath

Vice President, Product Management, UiPath

UiPath are a participating in EuroSTAR Conference 2026 as a Diamond Sponsor

Agentic testing for the enterprise: Ushering in a new era of software testing

April 14, 2025 by Aishling Warde

In today’s fast-paced world of software development, the need for effective testing has never been more important. Conventional approaches to testing are often challenged by rapid release cycles and complicated integrations, leading to slow delivery, high costs, and low software quality. Agentic testing is an innovative way to address these challenges. This transformative approach to software testing empowers testers with advanced AI capabilities, allowing them to automate much broader, more non-deterministic, and non-linear efforts in testing. AI agents take on the tedious and time-consuming tasks, providing a level of productivity that is not possible with conventional testing methods.

So how can you get started with agentic testing? We’re bringing agentic testing to life with the launch of UiPath Test Cloud, now generally available. UiPath Test Cloud—the next evolution of UiPath Test Suite—equips software testing teams with a fully featured testing solution that accelerates and streamlines testing for over 190 applications and technologies, including SAP®, Salesforce, ServiceNow, Workday, Oracle, and EPIC. Let’s take a closer look.

Comprehensive testing capabilities for the enterprise

Test Cloud is an environment where software testers feel at home. It’s your solution for bringing agentic testing to life—augmenting you with AI agents across the entire testing lifecycle. Zooming in, Test Cloud is a fully featured platform designed to serve all your testing needs. Whether it’s functional or performance testing, Test Cloud empowers you with open, flexible, and responsible AI across every stage, from test design and test automation, to test execution and test management. And it’s built for scale—with everything you need to handle the largest and most complex testing projects. It helps you design smarter tests with capabilities like change impact analysis and test gap analysis, ensuring a risk-based, data-driven approach to testing. It gives you the flexibility to automate tests the way you want, whether it’s low-code or coded user interface (UI) and API automation, across platforms. Plus, with continuous integration and continuous delivery (CI/CD) integrations and distributed test execution, Test Cloud seamlessly fits into your ecosystem while accelerating your testing to keep up with rapid development cycles. And when it comes to test management, Test Cloud has you covered with 50+ application lifecycle management (ALM) integrations, as well as a rich set of test data management and reporting capabilities.

Unlock built-in and customizable AI for the entire testing lifecycle with UiPath Autopilot™ for Testers

What makes agentic testing truly agentic? AI agents. With UiPath Autopilot for Testers, our first-party AI agent available in Test Cloud, you’re equipped with built-in, customizable AI that accelerates every phase of the testing lifecycle.

Leverage Autopilot to enhance the test design phase through capabilities such as:

Quality-checking requirements
Generating tests for requirements
Generating tests for SAP transactions
Identifying tests requiring updates
Detecting obsolete tests

Then, use Autopilot to take your test automation to the next level through capabilities such as:

Generating low-code test automation
Generating coded user interface (UI) and API automation
Generating synthetic test data
Performing fuzzy verifications
Generating expressions
Refactoring coded test automation
Fixing validation errors in test automation
Self-healing test automation

And enhance test management with Autopilot capabilities such as:

Generating test insights reports
Importing manual test cases
Searching projects in natural language

Any type of tester—from a developer tester to a technical tester to a business tester—can use Autopilot to build resilient automations more quickly, unlock new use cases, and improve accuracy and time to value. Organizations are already yielding tangible benefits from this versatility and efficiency, as showcased by Cisco’s experience with Autopilot in accelerating their testing processes.

“At Cisco, our GenAI testing roadmap centers on leveraging UiPath LLM capabilities throughout the entire testing lifecycle, from creating user stories to generating test cases to reporting, while ensuring seamless integration with code repositories,” said Rajesh Gopinath, Senior Leader, Software Engineering at Cisco. “With the power of Autopilot, we’re equipped to eliminate manual testing by 50%, reduce the tools used in our testing framework, and reduce dependency on production data for testing.”

Build your own AI agents tailored specifically to your unique testing needs with Agent Builder

Now, let’s meet the toolkit for building AI agents tailored to your testing needs: UiPath Agent Builder. Leverage a prebuilt agent from the Agent Catalog, or build your own agent using the following components:

Prompts: define natural language prompts with goals, roles, variables, and constraints
Context: use active and long-term memory to inform the plan with context grounding
Tools: define UI/API automations and/or other agents that are invoked based on a prompt
Escalations: asks people for guidance with UiPath Action Center or UiPath Apps
Evaluations: ensure the agent meets your desired objectives and behaves reliably in various scenarios

Looking for inspiration to jumpstart your first attempt at building an agent? Here are some recommendations for agents that you can build to help accelerate your testing:

Data Retriever: helps find test data for exploratory testing sessions in databases
Bug Consolidator: identifies distinct bugs behind failed test cases after nightly test runs
Compliance Checker: finds test cases that do not adhere to best practice
Stability Inspector: identifies flaky tests, repeatedly failed tests, and false positives

These are just a few agents that augment your expertise throughout the testing lifecycle. Join the Agent Builder waitlist to be the first in line to try your hand at building one.

Open, flexible, and responsible

Beyond AI agents, what does Test Cloud offer that helps you engage in agentic testing?

With UiPath Test Cloud, you can harness the power of an open and flexible architecture that seamlessly integrates with your existing tools, including connections with your CI/CD pipelines, ALM tools, and version control systems, as well as webhooks that keep you informed in real time. This flexibility ensures that Test Cloud adapts to your unique enterprise needs.

When it comes to responsible AI, you benefit from the UiPath AI Trust Layer, which provides you with explainable AI, bias detection, and robust privacy protection. You can confidently meet regulatory requirements and internal governance standards thanks to comprehensive auditability features. By embracing the open architecture and responsible AI capabilities of Test Cloud, you’re not just streamlining your testing process–you’re future-proofing your software quality with intelligent, efficient, and trustworthy technology that grows with your team’s needs.

Resilient end-to-end automation

With UiPath Test Cloud, you can unlock the power of resilient end-to-end automation that will enhance your testing processes. Experience seamless automation capabilities for any UI or API, giving you unparalleled flexibility in your testing approach. Whether you’re working with home-grown web and mobile applications or complex enterprise systems like SAP, Oracle, Workday, Salesforce, and ServiceNow, you can engage in automated testing that covers all aspects of your software ecosystem. By leveraging powerful end-to-end automation, you’ll not only improve the efficiency of your testing processes but also gain greater confidence in the quality and reliability of your software releases. Customers like Dart Container, Quidelortho, Orange Spain, and Cushman and Wakefield have achieved 90% automation rates, 30-40% cost reduction, 6X faster release speeds, and other significant benefits through using UiPath automated testing capabilities.

Production-grade architecture and governance

You and your team may face the challenge of maintaining a secure, scalable, and compliant testing infrastructure that can keep up with your agile development processes. With Test Cloud, you’re equipped with a production-grade architecture and robust governance features that will transform your agentic testing experience.

Benefit from Veracode certification, ensuring your testing environment meets the highest security standards and giving you peace of mind. Comprehensive auditing capabilities provides you with detailed insights into all testing activities, enabling you to maintain full transparency and easily demonstrate compliance. You also have granular role management features, allowing you to precisely control access and permissions, ensuring that the right people have the right level of access at all times. With centralized credential management, you can streamline security processes and reduce the risk of unauthorized access, making it easier than ever to manage and protect sensitive testing data.

Powered by the UiPath Platform

When you choose UiPath Test Cloud, you’re not just getting a standalone testing solution–you’re tapping into the power of the entire UiPath Platform™. This opens up a world of possibilities for streamlining your testing processes and boosting your overall automation efforts. You’ll benefit from shared and reusable components across teams, allowing you to leverage expertise and reduce duplication of effort. EDF Renewables, for example, achieved 75% component reuse by leveraging testing capabilities within the UiPath Platform. With access to the UiPath Marketplace, you’ll have a wealth of prebuilt solutions at your fingertips, accelerating your testing initiatives. Access to snippets and libraries empowers you to create modular, reusable code that can be easily shared and maintained across your organization. Plus, you can leverage centralized object repositories, which simplify test maintenance and improve consistency across your automation projects. Additionally, the robust asset management capabilities ensure that you can efficiently organize, version, and deploy your automation assets enterprise-wide, maximizing the value of your organization’s investment in the UiPath Platform™.

The benefits of agentic testing with UiPath Test Cloud

No matter your role or ranking at your organization, you can start reaping the benefits of Test Cloud for agentic testing right away. As a CIO, you’ll experience increased efficiency, reduced costs, and better resource utilization, ultimately leading to faster time-to-market and enterprise-wide automation. Testing team leads will benefit from improved consistency and reliability, increased productivity, and better defect detection, while standardizing testing processes and achieving unprecedented scalability. For testers, Test Cloud offers increased accuracy and efficiency, enhanced test coverage, and faster feedback loops, resulting in higher job satisfaction. The tangible benefits are clear: based on an in-depth study conducted by IDC, customers using UiPath for testing have achieved $4M average annual savings per customer, 529% three-year return on investment, and 6 months payback on investment.

With agentic testing powered by Test Cloud, all roles will enjoy accelerated test cycles, deeper test coverage, and reduced risk, all while realizing significant cost savings and resource optimization. This comprehensive and adaptive testing approach will empower your organization to deliver high-quality software faster than ever before, accelerating your time to value and giving you a competitive edge in today’s fast-paced software landscape. This vision of AI-augmented testing is not just theoretical; forward-thinking organizations like State Street are already anticipating how it will transform their testing processes.

The future of agentic testing

Test Cloud isn’t just built for the testing you know today—it’s built for where testing is going. With Test Cloud, you’re not just keeping up with increasing testing demands—you’re staying ahead. Get started with UiPath Test Cloud by signing up for the trial today.

Author

Sophie Gustafson

Product Marketing Manager, Test Cloud, UiPath

Real-World Data vs. Fake Data: Choosing the right strategy for effective testing

April 1, 2025 by Aishling Warde

Testing environments play a critical role in software development, ensuring applications function correctly before release. To achieve this, having test data that simulates real-world scenarios is essential. However, the choice between “fake data” and “real-world data” sparks an interesting debate, as each approach offers significant challenges.

In this article, we will explore the key differences between these two types of data, analyze their benefits and challenges, and ultimately highlight how a strategic combination of both can optimize the testing process, ensuring accuracy, security, and efficiency in development environments.

What is real-world data?

Anonymized real-world data is derived from production environments, ensuring it does not contain personally identifiable information while complying with regulations such as GDPR, CCPA, LPDP, and others.

These datasets offer a high degree of realism, as they preserve referential integrity, maintain the natural complexity of real-world scenarios, and accurately reflect user behavior, system interactions, and business logic. Additionally, real-world data naturally exhibits aging, reflecting how information changes over time and capturing historical trends and patterns that influence system behavior.

By leveraging real-world data, organizations can test applications under conditions that closely resemble actual usage, improving the reliability and effectiveness of their testing processes.

What benefits do real-world data offer?

Using real-world data provides significant advantages for your organization:

Captures the complexity of real-world behavior, including intricate patterns, sudden fluctuations, and inherent biases while ensuring data privacy.
Maintains appropriate statistical distribution and frequency.
Preserves relationships and interdependencies between elements, allowing comprehensive “end-to-end” testing.
Reduces the gap between development, testing, and production environments.
Facilitates integration testing with other systems under production-like conditions.
Provides immediate availability and reusability.

Challenges of using real-world data

Working with anonymized real-world data presents challenges. Identifying the right data for each test case, anonymizing it effectively, and delivering it on-demand to the testing environment are key challenges, especially in complex and costly environments with large volumes of data. Managing real-world data requires robust tools to ensure that no sensitive information is exposed and that masking processes remain effective, as well as addressing other critical challenges in test data management.

Synthetic data

The term “fake data” or “synthetic data” is widely used across industries but lacks a universally accepted definition. Different sectors and vendors interpret this concept in various ways depending on their testing needs and available technologies. While some consider synthetic data as manually created datasets, others define it as AI-generated data, or even simply masked real data. As these variations can create confusion, understanding the most common approaches provides greater clarity about what synthetic data really means.

Some of the most common definitions include:

Traditionally created data: Data manually or traditionally generated using tools like spreadsheets, scripts or bussines apis. While quick to produce, it often lacks complexity, is prone to errors, and becomes costly over time.
AI-Generated data: Data created by AI models trained on real-world patterns. Although it can mimic realistic behaviors, its reliability remains limited for mission-critical applications. For the time being, there is no evidence of successfully using this approach for testing business support systems.

Synthetic data limitations

These approaches to synthetic data generation often fall short when it comes to accurately simulating production environments, facing critical limitations challenges such as:

Lack of aging: No representation of time-based changes.
Limited complexity: Misses intricate, real-world dependencies.
Absence of rare scenarios: Struggles to simulate edge cases.
No technical debt: Fails to reflect legacy patterns and old system quirks.
Unrealistic data: Lacks inconsistencies found in production.
Reduced data richness: Missing the diversity of real-world interactions.
Insufficient volume: Smaller datasets than real production environments.
Inaccurate data distribution: Does not replicate real-world patterns.

These gaps make these synthetic data approaches unreliable for testing environments that aim to mimic production conditions accurately.

How does icaria Technology generate high-quality synthetic data?

To overcome these limitations, icaria Technology has developed a model-based synthetic data approach that ensures realistic, secure, and scalable datasets for high-quality testing environments. This approach allows us to create high-quality test data that mirrors real-world conditions without compromising security, compliance, or performance.

Advantages of icaria Technology’s synthetic data

Our approach to synthetic data offers significant advantages for software testing environments. By replicating the structure, patterns, and complexity of real-world data while ensuring the exclusion of sensitive information, this method strikes a balance between realism, scalability, and security. Here are some key benefits of using our synthetic data:

Realistic test scenarios with no privacy risks
Maintains relationships, distributions, and behaviors from real-world datasets without exposing PII. By generating this data from pre-existing models, we ensure that test environments mirror production scenarios.
Consistency across testing stages
Ensures smooth transitions between development, staging, and production phases by preserving referential integrity and data relationships.
Scalability and flexibility
Generates large volumes of test data tailored to specific needs, supporting extensive performance and scalability tests.
Customizable for testing requirements
Allows the generation of datasets designed for edge cases, rare scenarios, or new application features.
Cost efficiency
Reduces manual effort and minimizes rework costs through automated processes, ultimately saving resources during the testing lifecycle.

When to use real-world data and when synthetic data?

After reviewing what real-world data is and our definition of synthetic data, the question arises: which one should we use in testing?

Real-world data is the best option for testing due to its richness and complexity, accurately reflecting system behavior and user interactions. Since this data already exists, it is often more efficient to use it rather than generating new datasets, which can introduce additional challenges and complexities.

However, this does not mean synthetic data has no place in a robust testing strategy. In certain situations, our synthetic data approach can be particularly useful, such as:

When testing requires data that is not yet available in existing application environments. For instance, during new developments involving changes to the application’s data model, there will be no existing data for the new model, necessitating synthetic data generation.
When specific datasets are rare but essential for testing. Some scenarios occur infrequently, meaning only one or two real-world examples exist. In these cases, synthetic data can generate additional instances, ensuring all testers and developers have access to the necessary data.

The perfect combination for reliable testing with icaria TDM

In the high-complexity environments managed by icaria Technology, particularly in icaria TDM, the reality is significantly more complex. These applications function in mission-critical domains where the margin for error is nonexistent.

By combining real-world data with synthetic data, organizations can create a balanced and efficient approach to test data management that ensures accuracy, compliance, and scalability.

Choosing the right type of data for each scenario, or combining both, helps companies improve test quality, comply with regulations, and optimize resources. With icaria TDM, achieving this balance has never been easier. This approach not only enhances testing efficiency but also strengthens confidence in systems, ensuring applications meet the highest quality standards before deployment.

Author

Enrique Almohalla

Enrique Almohalla, leading icaria Technology as CEO, brings a wealth of experience in TDM methodologies, cultivated through over twenty years of directing software development and testing projects. His significant involvement in Test Data Management, marked by continuous innovation and application, underscores his deep understanding of the field. Additionally, his position as an associate professor at IE Business School in Operations and Technology melds his hands-on experience with academic insights, offering a comprehensive perspective on business management.

iCaria were Exhibitors in EuroSTAR 2025. Join us at EuroSTAR Conference in Oslo 15-18 June 2026.

Understanding Model-Based Testing: Benefits, Challenges, and Use Cases

March 17, 2025 by Aishling Warde

For test engineers seeking a systematic and organized approach to testing, model-based testing offers a powerful toolset. This method involves working with models that guide the testing process.

Besides creating models of tests, you can model, for example, application behavior, application structure, data, and environment. In this article, our core focus will be on testing – so, thinking about what aspects to test and how to do that drives the modeling.

Let’s delve deeper into what model-based testing entails, its benefits, challenges, and scenarios where it is most effective.

What is Model-Based Testing?

Model-based testing is a testing approach that revolves around the use of models. Unlike traditional testing, which involves scrutinizing every intricate detail, model-based testing takes a more general approach. It allows you to concentrate on the core functionalities without being bogged down by all the little details.

Let’s take an example – say that you’re testing an address book application. In this case, you could model the following actions:

• Start the application
• Create a new file
• Add contacts
• Remove contacts
• Save the file
• Open the file
• Quit the application

The idea is not to model the whole application, as a developer would, but rather to get a grasp of the test cases you need to prioritize. This will help in organizing your test cases and in the end your test scripts, which can then be used for automating the test cases.

Benefits of Model-Based Testing

Helps focus on the things that matter
By focusing on high-level abstractions, model-based testing helps you avoid getting lost in the details. This strategic approach allows you to skip unnecessary test cases, optimizing testing efforts and resources.

Ultimately, this leads to higher-quality tests that accurately represent critical functionalities.

Makes communication easier
Models help in finding a common understanding of the requirements and detecting potential misunderstandings. They make it easier to convey testing needs to both internal and external stakeholders.

For example, with models, you could show the management what your test process looks like and why additional resources are needed. Or you could explain to the development team how you’re currently testing and discuss why something is not working as it should.

The visual aid that models offer is often more effective than discussing the problems verbally or looking at abstract test scripts.

Better communication in the early stages of the development process also leads to early detection of bugs – our benefit number 3.

Avoid defects in the early stages of the product
In the traditional development process, the steps of requirements, design, and testing are performed sequentially using a variety of tools. As testing is the final stage, most defects – accumulated throughout the previous stages – are caught quite late in the process. This makes fixing them time-consuming and costly.

Model-based testing is one methodology further enabling so-called shift-left testing. This refers to the shift in the timeline – testing can begin already at the requirement phase.

Models can be shared with project stakeholders, before the implementation, to verify requirements and to identify gaps within the requirements. It might also reveal a problem area if you cannot model something.

As a result, defects are caught and removed earlier, lowering the total cost of development. According to MathWorks, the savings can range from 20 to 60% when compared with traditional testing methods.

Effort reduction in implementation and maintenance

While modeling requires initial effort, it significantly reduces the effort needed for implementation and maintenance.

Model-based testing utilizes the modularization of test cases. In the case of traditional testing, when some element of your application changes, you might have to change every individual test case. With model-based testing, you can use the building blocks, like Legos, and fixing one single block will bring all your test cases up to date.

Also, there are time-saving benefits as you learn to operate in a more organized way. You can detect the highest priority tests – and avoid any redundant work.

Challenges of Model-Based Testing

Mindset transition

Transitioning from a traditional testing process to model-based testing requires a period of adjustment and learning.

Specific skill set required

Not all test engineers may be proficient in abstract modeling. Creating effective models demands skills such as abstract thinking and generalization. To succeed, you need to keep a bird’s eye view of the whole testing process.

Abstraction level challenge

Selecting the right level of abstraction is crucial. Too abstract, and tests may become less useful; too detailed, and the model may be challenging to work with.

However, abstraction inherently involves simplification and can lead to the loss of critical details, potentially overlooking important aspects.

When to Choose Model-Based Testing?

While model-based testing is a powerful tool, it may not be suitable for every scenario. If you’re dealing with a straightforward application, it may be overkill, potentially leading to over-engineering.

However, for complex software systems and teams capable of working at abstract modeling levels, model-based testing proves invaluable.

Conclusion

Model-based testing is a powerful approach that empowers test engineers to focus on testing the critical aspects of the application under test. By leveraging models as high-level abstractions, teams can enhance test quality, reduce effort, and improve communication.

While it requires a shift in mindset and specific skills, the benefits far outweigh the challenges, particularly in complex software environments. As with any testing methodology, the key lies in thoughtful application and adaptation to suit specific project needs.

In the second part of this article we dive into model-based testing best practices and testing tools. Here you will find a real world example on how to achieve model-based testing in Squish.