Software Testing

Gen AI for Software & Quality Engineering – Elevate your possible

May 26, 2026 by Lauren Payne

When we look at the history of software engineering and quality engineering (QE), we often see significant shifts occurring every decade or so. Each has brought with it new tools, processes and methodologies that have improved the way we develop, test, and maintain software applications, and by extension, the experiences we deliver to end users.

Test automation emerged in the 1970s to accommodate growing business demands, with Agile methodologies following in the ’90s to enhance quality and security. From the 2000s, DevOps and low-code strategies enabled quicker deployment and talent attraction.

The software industry was further propelled by social, mobile, analytics, and cloud (SMAC) technologies, completely transforming it as we knew it.

Despite these advancements, challenges persist in delivering quality products at speed, managing cybersecurity risks, containing costs, reducing technical debt, and sourcing skilled talent.

Now, with the emergence of generative AI, we stand on the brink of the next major evolutionary leap in software and quality engineering. This technology merges with traditional engineering principles to significantly expedite development, testing, and maintenance processes—ensuring faster market delivery of innovative products while freeing up time for strategic initiatives.

Introducing the Gen AI Amplifier for Software and Quality Engineering: Quality Reimagined.

Accelerate QE with the Gen AI Amplifier

The Gen AI Amplifier for Software and Quality Engineering is a groundbreaking accelerator designed to optimize the Quality Engineering & Testing (QE&T) stages of the software development life-cycle (SDLC). By applying the powerful capabilities of generative AI to our industry-leading best practices, frameworks, and methodologies, we’ve crafted a uniquely amplified approach.

This accelerator is tailored to enhance the efficiency of QE&T tasks across the entire software life-cycle, from planning and design to building, testing, and deployment. It incorporates pre-built use cases and AI-driven prompts for every life-cycle stage—ranging from requirements and user stories to architecture modeling, design, code transformation and test case generation, API testing, synthetic test data and test automation.

The Gen AI Amplifier is built on a robust QE foundation using our proprietary assets, and thoroughly tested and validated to ensure consistent, reliable results. It is integrated with multiple LLMs within a secure architecture framework, which is further strengthened by our gen AI guardrails, knowledge framework, and data platforms.

The Gen AI Amplifier is helping us do in minutes and hours what software development and QA teams have traditionally needed days or weeks for:

Generating first requirements from conversation documents
Generating TMAP-guided test cases from requirements
Generating comprehensive test data
Generating automated test scripts, and more.

Generative AI adoption is driving impact across the entire test ecosystem

In the 15th World Quality Report, we learned that out of the 1,750+ senior executives surveyed globally, 64% have identified processes or applications that can benefit from AI. The majority are using AI towards building and improving test scope along with improving performance engineering, as well as the test ecosystem overall. They are actively utilizing AI to optimize their testing processes. 79% of the quality leaders agree that AI systems are going to be used to help them optimize their test scope and increase velocity.

64% of organizations have identified processes/ applications that can benefit from AI.

79% of quality leaders agree that AI systems are going to be used to help them optimize their test scope and increase velocity.

(Data from World Quality Report, 15th Edition, 2023-24)

The Gen AI Amplifier supports the full life-cycle for agile teams. We’ve integrated multiple Gen AI models in a secure environment, with pre- engineered and tested prompts built with our industry-leading quality engineering methodology, agile framework, and cloud-native architectural best practices.

With the help of our AI experts at Sogeti, the Gen AI Amplifier augments business analysts, product owners, lead architects and quality engineers at every stage – from requirements gathering and infrastructure design to comprehensive software testing.

Quality amplified early on, and at every stage, yields a significantly increased level of productivity, efficiency, and acceleration.

Closing thoughts, for now

Diving into generative AI is really exciting, and I believe in the importance of remaining grounded and focused on pragmatic experimentation. This isn’t just about incremental improvements—it’s about redefining our approach to software engineering and QA. While our ultimate goals haven’t changed, our means to achieve them have evolved. Let’s explore generative AI’s potential responsibly, learning and adapting as we go. Together, we’ll discover how these advanced tools can enhance our industry without losing sight of our core objectives.

Watch this space as we unveil more about the Gen AI Amplifier and its impact on software QA.

Author

Antoine Aymer

CTO for Quality Engineering & Testing, Sogeti

Sogeti are Exhibitors in EuroSTAR 2026. Join us at EuroSTAR Conference in Oslo 15-18 June 2026.

Testing AI Agents: A Practical Blueprint for Custom Evaluation Frameworks

May 13, 2026 by Lauren Payne

A Leader’s guide to building domain-aware evaluation disciplines that turn experimental AI pilots into production-grade, auditable enterprise systems.

The AI Agent Production Gap

AI agents are moving rapidly into software engineering, testing, DevOps, support, and back-office workflows. Many organisations have running pilots; far fewer trust those agents enough to put them into production. Gartner predicts that more than 40% of agentic AI projects may be cancelled by the end of 2027 , citing cost, unclear business value, and inadequate risk controls.

The blocker is rarely the model. It is the absence of an evaluation discipline that can answer a harder question: can this agent complete the right task, in the right context, with the right controls, consistently?

Why AI Agents Need a Different Testing Strategy

Traditional software testing assumes a known input, a fixed expected output, and repeatable behaviour. AI agents do not work that way. Outputs vary. Retrieval can return different chunks. Tool calls may take different paths. The reasoning trace shifts from one run to the next.

Evaluating an agent requires checking whether it understood the user’s intent, retrieved the right context, called the right tool, avoided hallucination, followed enterprise policy, and escalated when its confidence was low. No single generic metric covers all of this.

Existing Evaluation Frameworks Are Necessary, Not Sufficient

A strong ecosystem of evaluation tools already exists. DeepEval, Ragas, Promptfoo, LangSmith, Braintrust, TruLens, Phoenix, and OpenAI Evals each give teams real leverage on prompts, RAG pipelines, model outputs, hallucination, retrieval quality, tool calls, traces, and regression behaviour. They are essential building blocks.

But a customer-support agent, a banking-compliance agent, a test-case generation agent, and a Playwright automation agent may all share an LLM core while having entirely different definitions of “good.” Generic accuracy and faithfulness scores cannot decide whether a generated test suite is release-ready. The evaluation tool provides the engine; the organisation must define the quality model

The Custom Evaluation Blueprint

A practical custom evaluation framework can be built in seven steps:

Define the agent’s mission Write down what the agent must do, what it must never do, and how much autonomy is allowed before a human is required. This becomes the evaluation contract.
Build task-level evaluation datasets Cover normal flows, edge cases, negative scenarios, ambiguous prompts, high-risk domain cases, and historical production issues.
Create domain-specific rubrics Score domain relevance, business accuracy, retrieval correctness, reasoning, tool correctness, hallucination control, compliance, clarity, and escalation behaviour.
Apply weighted scorecards A formatting slip is low severity; a wrong business recommendation is critical; a wrong tool call may block release. Weight accordingly.
Combine automated evaluation with human calibration Automated evaluators give scale; expert reviewers calibrate the rubric over time to account for edge cases no automated scorer anticipated.
Run regression evaluation continuously Re-score whenever the model, prompt, RAG corpus, tool definition, workflow, or enterprise policy changes.
Convert scores into release gates Pass · Conditional Pass · Human Review Required · Block — each gate tied to a clear business risk threshold.

Custom Metrics Based on Business Context

Generic LLM benchmarks measure model capability in isolation. Enterprise AI agents operate in a business context — with specific user personas, data governance requirements, integration constraints, and financial consequences of failure. The metrics must reflect that context. Below is a framework for selecting and weighting evaluation dimensions by deployment domain.

Metric Clusters by Enterprise Domain

Each domain cluster below contains the metrics that carry the most signal for that type of agent. Select the cluster that matches your deployment, then tune weights using your organisation’s risk tolerance and regulatory posture.

Weighted Scorecard: Enterprise AI Agent Release Template

The table below shows how to structure a weighted scorecard across core evaluation dimensions. Adjust weights to match your domain cluster above and your organisation’s risk posture.

Business-Context Metric Matrix

The following matrix maps enterprise agent types to their primary KPI, the hardest-to-catch failure mode, and the metric that most reliably surfaces it.

Release Gates — Translating Scores into Decisions

Every weighted scorecard must terminate in a binary business decision. The four-gate model below maps score ranges to actions and assigns responsibility for each outcome.

Lessons from testron.ai Implementations

What This Means for QA Teams

Agentic AI is reshaping the QA mandate. Test execution is no longer the centre of gravity; evaluation design is. The QA function becomes the quality gatekeeper for enterprise AI agents — owning rubrics, scorecards, regression datasets, and human-in-the-loop calibration.

The skills that compound from here are domain-aware evaluation design, structured human review, and translating business risk into release gates that engineering and the business both trust.

Closing

The future of testing is not just more automation. It is trusted AI-agent evaluation: a clear mission, a custom rubric tuned to business context, a weighted scorecard, calibrated human review, and a release gate that reflects business risk.

Teams that build this discipline now will be the ones who put agents into production with confidence — and who earn the trust of the board, the regulator, and the customer.

References:
Gartner: Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 — gartner.com
DeepEval — deepeval.com

Author

Babu Manickam CTO, Indsafri

Over 27 years of experience in software testing, test automation, performance engineering, DevOps, and AI-led quality engineering. He has trained more than 50,000 QA professionals and works with enterprises to implement modern testing practices across automation, Generative AI, and agentic quality engineering. An active speaker and community contributor in the software testing ecosystem, with a strong focus on helping QA professionals transition into AI-augmented engineering roles.

Indsafri are Exhibitors in EuroSTAR 2026. Join us at EuroSTAR Conference in Oslo 15-18 June 2026.

The Real Problem with AI Testing Tools Isn’t the AI – It’s Trust

May 11, 2026 by Lauren Payne

Every software company is talking about AI. Copilots. Autonomous agents. Self-healing tests. But inside most engineering organisations, a quiet hesitation persists – not because teams doubt AI’s potential. Precisely because they do.

The real question holding teams back isn’t capability. It’s a far more fundamental one: Can we trust AI to participate in release-critical testing workflows?

That question matters because software testing is fundamentally different from most other AI-assisted tasks. A poor AI-generated image is an inconvenience. A poor AI-governed production release can damage revenue, customer trust, compliance, and operational stability.

The next evolution of quality engineering will not be driven by fully autonomous AI in isolation. It will be driven by human-controlled Agentic AI.

The Industry Has a Scaling Problem

Most organisations already have automation. What they struggle with is scaling it.

Traditional automation frameworks demand specialist skills, constant maintenance, brittle selectors, and significant engineering effort just to keep regression suites operational. As applications evolve, teams often spend more time maintaining tests than expanding meaningful coverage.

Meanwhile, release velocity continues to accelerate. Modern engineering teams are expected to deliver continuously across web, mobile, APIs, and increasingly complex customer journeys. The operational cost of maintaining automation has quietly become one of the biggest hidden blockers in software delivery.

This is where Agentic AI changes the conversation entirely.

From Scripts to Intelligent Testing Systems

At Virtuoso, our Agentic AI vision is built around a single, clear outcome:

Rather than treating test automation as a collection of static scripts, Agentic AI introduces intelligent operational workflows that help QA teams move from requirements to execution dramatically faster. The platform is designed to:

But critically, AI does not act unchecked. Every important decision point remains reviewable, traceable, and human-governed. Requirements are approved. Generated journeys are reviewed. Repairs are auditable. Changes are versioned.

The Future Is Not AI Replacing Testers

One of the most persistent misconceptions surrounding AI in testing is that the goal is complete removal of human involvement. In reality, experienced QA engineers are becoming more valuable – not less.

The organisations seeing the greatest success with AI are those combining machine scale with human judgement. That is the core philosophy behind Virtuoso’s approach to Agentic AI.

Operational QA, Not Just Better Automation

The industry has spent years focused on “test automation tools.” The next phase is operational QA systems – systems that can understand context, coordinate workflows, propose actions, and continuously improve automation at a scale that would traditionally require large engineering teams.

But operational credibility only comes from governance. Engineering leaders don’t simply need AI that can generate tests. They need AI they can trust inside release-critical pipelines. That means:

This is precisely why human-controlled Agentic AI will ultimately outperform uncontrolled autonomy in enterprise software delivery.

The Next Era of Quality Engineering

AI will absolutely transform software testing. But the winners in this space will not be organisations chasing fully autonomous systems with no oversight. The winners will be teams that successfully combine:

AI-Powered Scale
Intelligent Orchestration
Human Governance
Operational Trust

In software delivery, speed matters. But trust matters more.

Author

Andy Dickin – QA & Quality Engineering Leader Virtuoso QA

Andy specialises in AI-powered automation and modern quality engineering practices that help organisations scale software delivery with confidence.

Virtuoso are Exhibitors in EuroSTAR 2026. Join us at EuroSTAR Conference in Oslo 15-18 June 2026.

Meeting the AI Software Quality Challenge: SmartBear’s Testing Portfolio

April 27, 2026 by Lauren Payne

AI-accelerated development has fundamentally changed how software is built, and across the industry, its impact on quality is already measurable. In SmartBear’s Closing the AI software quality gap study, we found nearly 70% of software professionals report application quality is declining as AI speeds up code generation, with development velocity increasingly outpacing teams’ ability to test effectively.

This is not a future risk or a theoretical concern. The gap between code generation speed and testing capacity continues to widen, creating an unsustainable dynamic. Teams face an impossible choice: either bottleneck development to maintain testing rigor or accept degraded application quality as development races ahead unchecked. But what if that tradeoff isn’t actually necessary?

Application integrity: The new standard for AI-era quality

Application integrity is the continuous, measurable assurance that your software works as intended at AI speed and scale. When code generation wildly outpaces application validation, maintaining integrity becomes impossible without sacrificing the velocity gains that AI-accelerated development promises. The consequences of compromised application integrity are severe: regulatory fines, brand damage, customer loss, and revenue impact. SmartBear addresses this challenge with the SmartBear Application Integrity Core™ – unifying the system of record (API catalog and test repository) with MCP tools and agentic workflows that empower developers and AI agents to deliver software that just works. Whether testing runs in cloud-native environments, on-premises infrastructure, or is managed directly within Jira, teams can continuously validate applications while maintaining control over quality as development accelerates.

SmartBear’s testing portfolio: Built for AI velocity

BearQ™: Autonomous QA for the next generation

SmartBear BearQ represents a fundamental shift in how testing keeps pace with AI-accelerated development. This agentic QA system operates at the highest levels of autonomy, serving as an exploration and testing analog to autonomous coding tools. BearQ thinks and tests like a real user, exploring applications autonomously and discovering flows rather than following pre-determined scripts. It adapts continuously as applications evolve, automatically updating tests without manual rewrites while maintaining full human visibility and control.

Reflect: Vision-based AI automation for modern applications

Reflect is a cloud-native test automation platform that uses vision-based AI to create and maintain tests that remain stable as applications evolve. By interpreting the UI the way users do, Reflect removes dependency on brittle selectors, enabling automation across web, mobile, and API workflows within a single platform. Teams can generate tests agentically or through natural language prompts, with built-in self-healing that automatically adapts to UI changes, reducing maintenance overhead while expanding coverage.

TestComplete: Enterprise desktop and web UI automation

TestComplete provides deep automation support for complex desktop applications, internal web systems, and legacy frameworks that modern cloud-first tools cannot reliably support. Its ability to run in secure, on-premises environments makes it essential for organizations with compliance requirements or specialized UI frameworks. Supporting multiple automation approaches – from record-and-replay to full scripting – TestComplete enables teams with different skill levels to work within the same system. Advanced hybrid object recognition combines property-based detection, text extraction, and vision AI to interact accurately with complex interfaces.

QMetry: Enterprise testing platform for scalable QA

QMetry is an enterprise test management platform that unifies performance, visibility, and automation in a single system designed to handle millions of test cases without performance degradation. As a centralized testing system of record, QMetry provides real-time visibility, audit-ready traceability, and customizable reporting across the entire organization. AI-driven capabilities streamline test creation and maintenance, with automated test case generation reducing creation time from 30-60 minutes to under 60 seconds. Built-in compliance features support regulated environments with flexible deployment options.

Zephyr: Jira-native testing for agile teams

Zephyr integrates testing directly within Atlassian Jira workflows, enabling teams to create, execute, and track tests alongside user stories, requirements, and defects without switching tools. This Jira-native approach provides end-to-end traceability across planning, execution, and validation while maintaining performance even as test libraries grow. Rovo agent skills enable natural-language queries to evaluate test coverage and assess release readiness, while MCP server capabilities extend Zephyr beyond Jira for more flexible workflows.

Swagger: Spec-driven API testing and contract validation

Swagger enables teams to design, test, document, and govern APIs using OpenAPI as a shared source of truth. By deriving testing directly from API specifications, Swagger reduces drift between design and implementation while enabling both functional validation and contract testing. Swagger Functional Testing validates endpoints against OpenAPI specifications, ensuring requests, responses, and data structures conform to defined contracts. Swagger Contract Testing verifies that API changes don’t break downstream consumers, critical for distributed and microservices-based architectures.

ReadyAPI: Comprehensive API testing for real-world conditions

ReadyAPI enables teams to validate API behavior across functional and performance scenarios while simulating dependencies through service virtualization. Supporting REST, SOAP, GraphQL, JMS, and other protocols, this on-premises platform allows functional tests to be converted into load tests without rebuilding scenarios. LLM-driven test generation creates and validates complex test cases with large data volumes using no-code, prompt-based workflows. Service virtualization simulates dependent systems, enabling testing when external services are unavailable – especially valuable in complex environments requiring infrastructure control.

A testing system that scales with modern development

The SmartBear testing portfolio addresses the fundamental challenge facing development teams: maintaining application integrity as AI accelerates code generation. Individual tools solve specific testing challenges across UI automation, API validation, and test orchestration. Together, they create a unified testing system that scales with AI-driven development velocity.

When testing infrastructure operates as a coordinated system rather than isolated tools, teams gain the ability to validate applications comprehensively without sacrificing speed. Automation scales without becoming fragile. API changes are validated before reaching consumers. Testing coverage remains aligned with development rather than trailing behind it. The result is not a choice between speed and quality – it’s the ability to deliver both while maintaining the application integrity that modern software demands.

Author

Rob McNeil Senior Manager of Product Marketing

Rob is a Senior Manager of Product Marketing focused on defining the go-to-market strategy for SmartBear’s portfolio of software testing products. He is passionate about engaging with customers and bringing their voices into product strategy so that feature launches align with real market needs. He has been with SmartBear for four years, with his more recent projects centered on researching the impact of AI, including how it is bringing significant changes to software developers and testers, and launching new generative AI and agentic AI features to meet the demands of the evolving development landscape.

SmartBear are Gold Sponsors in EuroSTAR 2026. Join us at EuroSTAR Conference in Oslo 15-18 June 2026.

The AI Velocity Paradox: Why Your Test Data is the New Bottleneck

April 22, 2026 by Lauren Payne

Generative AI has dramatically accelerated software development. Ideas that once took weeks to turn into working code can now be prototyped in hours. AI tools generate code, suggest tests, analyze logs, and help engineers iterate faster than ever. According to McKinsey’s research on developer productivity, AI-assisted teams can complete coding tasks significantly faster, with some organizations reporting development velocity improvements of 30 to 40% in AI-supported workflows.

But something interesting is happening in many organizations: even as coding accelerates, releases still stall. Teams wait not for code to be written, but for confidence that it is safe to ship. In many cases, the blocker is not the tests themselves. It is the data behind them. In the AI era, test data has quietly become the new critical path to quality.

Consider a typical enterprise team using AI coding assistants. Development velocity improves immediately. Features move from idea to code faster than ever. But releases still slip because QA teams must wait days for masked datasets or environment refreshes. The development bottleneck is gone, but the testing bottleneck remains.

The Bottleneck is Shifting in the SDLC

AI now supports multiple stages of the software lifecycle, reducing work that once took days to mere hours. Yet validation still takes time. Teams can generate tests quickly, but they often wait on environments, data provisioning, and compliance approvals before those tests can run.

As development accelerates, constraints move downstream toward validation. Inside QA, test data and test environments are where projects most often slow down.

Why Test Data is Uniquely Hard

Accessing usable test data is rarely simple. Industry research consistently shows that testers spend up to 30 to 40% of their time searching for or preparing data rather than executing tests. Manual provisioning of test datasets can take days or weeks. That cadence does not match modern CI/CD pipelines or AI-assisted development.

When teams cannot access realistic data, test coverage suffers. Edge cases go untested, business workflows behave differently than they do in production, and defects slip through into live environments. These inconsistencies make tests unreliable and environments unpredictable. Furthermore, if automation frameworks and AI-driven testing tools rely on incomplete data, the automation itself becomes unreliable.

The AI Twist: More Code, More Testing Pressure

Agentic AI will generate more code changes and more test cases than any previous tooling. This increases the validation volume required in every build.

If teams run AI-generated tests on unrealistic datasets, the results can be highly misleading. A build may pass in the lab while still hiding defects that appear in production. “Green builds” do not always mean safe releases. Without production-like data, testing becomes a simulation detached from reality.

Why Traditional Test Data Management Breaks Under AI Velocity

Feature	Traditional Test Data Management	Modern Test Data Management (AI Era)
Provisioning Model	Centralized teams, ticket-based requests	Self-service, on-demand generation
Data Source	Full database copies (heavy, slow)	Multi-source data automatically masked, blended, and pushed in QA/UAT environments
Speed	Days, weeks, or months	Minutes (self-service UI or integrated into CI/CD pipelines)
Architecture Fit	Monolithic legacy systems	Spans legacy, SaaS, and micro-services
Compliance / PII	Manual masking that differs by database, inconsistent enforcement	Automated, centralized PII masking and synthetic data, built-in by design

What Good Test Data Looks Like Today

To support modern testing, data must meet three strict criteria:

Realistic: Datasets must reflect production behaviors, edge cases, and real business workflows.
Compliant: Personally identifiable information (PII) must be protected. Using synthetic test data (realistic but fictitious datasets generated to mirror production patterns) and advanced masking techniques helps preserve useful data characteristics without exposing real user details.
Consistent: Enterprise applications span multiple systems. Test data must reflect accurate relationships across services, platforms, and applications.

Platforms like K2view approach this by organizing enterprise data around business entities such as customers, policies, or orders using patented Micro-Database technology. This allows teams to provision complete, consistent datasets that reflect real user journeys across multiple systems while maintaining referential integrity. Instead of stitching together multiple tools, teams combine data masking, cloning, and synthetic generation within one platform.

Don’t Let Test Data Become the New Bottleneck

The bottleneck has moved, but it hasn’t disappeared. Teams that recognize test data as infrastructure — not an afterthought — will be the ones who actually realize the promise of AI-accelerated development. The rest will keep explaining why fast code still means slow releases.

Author

Amitai Richman, Director of Product Marketing K2view Test Data Management

Amitai Richman is Director of Product Marketing at K2view, where he leads GTM strategy for enterprise AI and test data management solutions. He specializes in translating complex data and agentic AI capabilities into clear business value for enterprise buyers, with a particular focus on TDM, synthetic data generation, and AI-ready data infrastructure. Amitai has presented on AI in software testing at industry forums across Europe and the US, and is a recognized voice in the product marketing and QA communities.

Scaling Human Evaluation of AI-Infused Applications

April 22, 2025 by Aishling Warde

It seems like everyone in the quality engineering community is talking about AI. After all, advances generative AI have been rapidly transforming our tools, frameworks, platforms, processes, and ways of working. AI-assisted software lifecycle activities are becoming more widespread and generally accepted.

However, while everyone has been busy applying AI to testing, myself and the team at Test IO have been focused on harnessing the collective intelligence of humans to validate and verify several different types of AI systems. Our approach is currently being utilized to test some of the most sophisticated AI models, assistants, co-pilots and agents at scale.

This article shares our approach to testing what we refer to as AI-infused applications. It describes the grand challenge with testing these types of applications and then discusses the need for human evaluation. After outlining a number of practical techniques, it provides some lessons learned on how to effectively scale the human evaluation of AI/ML.

AI-Infused Applications (AIIA)

Just as with any application, there are many different ways that an AI-based system can be implemented. The particular method used generally depends on the problem being solved, desired capabilities, and any constraints on its development and operation.

Some common approaches to developing AI systems are:

Rules-Based AI. Encodes domain expertise into conditional (if-then-else) statements, heuristics, or expert systems.
Classical Machine Learning. Training a supervised or unsupervised learning model from scratch using structured datasets and algorithms like random forests, support vector machines, or gradient boosting.
Integrating a Pre-Trained Model. Leverages APIs from providers like OpenAI, Google, Anthropic, or Hugging Face to integrate an AI-powered component into an application with minimal development effort.
Retrieval Augmented Generation. Combines LLMs with a vector search database to fetch relevant information before generating a response.
Fine-Tuning a Pre-Trained Model. Uses transfer learning to adapt a pre-trained model to a specialized dataset to improve the performance of specific tasks.
Agent or Multi-Agent Based AI. Builds autonomous agents using an LLM backend, goal-setting mechanisms, and tools for action execution, e.g., APIs, databases, browser automation. These types of systems may include reinforcement learning (RL) or emergent behavior to allow multiple AI agents to interaction, collaborate, or compete in an environment.
AI-Orchestrated Workflows. Integrates multiple AI components using workflow orchestration tools such as LangChain, Haystack, or Airflow).

Each of these approaches have trade-offs in terms of accuracy, efficiency, cost, interpretability, among others. However, regardless of the approach used, as long as an application leverages AI or ML models or services as part of their logic, we consider it to be an AI-infused application.

Grand Testing Challenge

The rapid pace of growth in the AI space makes it particularly difficult to keep track of all of the different ways these types of systems can be implemented. However, irrespective of the development method, AI-infused applications present a grand challenge for software testing practitioners primarily due to their highly dynamic nature.

Dynamism

AI-infused applications exhibit different levels of dynamism depending on their purpose and capabilities. For example, there are dynamic aspects of predictive, adaptive, and generative AI systems which make them unpredictable, non-deterministic and, as a result, very difficult to test.

Predictive AI analyzes historical data to identify patterns and make forecasts about future events or outcomes. These types of systems evolve with data. In other words, the accuracy of predictions depends on continuously updated data and therefore, as new data arrives, retraining or fine-tuning the model helps improve its forecasts. Some predictive systems like stock trading algorithms process real-time data streams, modifying forecasts as conditions change.
Adaptive AI continuously learns from new experiences and environmental changes to modify its behavior and improve performance over time. Unlike traditional models, it evolves without requiring explicit reprogramming. For example, self-learning chatbots will personalize their responses over time. Systems like these are context aware and dynamically adjust based on real-world conditions. Adaptive AI can autonomously tweak its internal models and strategies to improve accuracy and efficiency over extended use.
Generative AI creates new content such as text, images, code, and music based on learned patterns from vast datasets. The same user prompts sent to a model can generate different responses. Generative models can refine outputs based on user feedback, style preferences, leading to evolving content quality. Model knowledge can be augmented with external sources via retrieval augmented generation, making the overall system highly flexible.

Adequately testing AI systems may involve a combination of pre- and post release testing, continuous monitoring, automated pipelines, adversarial testing, and human evaluation. These approaches help address quality challenges with AI systems including model drift, bias, fairness, uncertainty, output-variability, explainability, hallucinations, and more.

The Importance of Human Evaluation

While automated testing methods for AI systems help to monitor performance, human evaluation is essential to ensure AI aligns with real-world expectations. Here’s why human evaluation is critical and some techniques that can be applied in practice.

Why It Matters

In classical ML systems, automated accuracy metrics such as F1 scores don’t necessarily capture the real-world impact of predictions. Bias and fairness issues often require domain experts to identify potential harms and when it comes to explainability, although some tools provide insights, human judgement is generally needed to interpret them meaningfully. If an AI-infused application is going to interact directly with users, the system must be assessed for user experience and usability.

Adaptive AI systems can potentially start optimizing on the wrong objectives. For example, a common problem with recommendation systems is that they tend to reinforce their own biases. Here’s how:

Recommendation system suggests content based on past user behavior.
User engages more with that types of content (e.g., specific movie genres, political articles)
System interprets this as a strong preference, leading it to narrow future recommendations to similar content.
Over time, diversity in recommendations decreases and users are less likely to be exposed to alternative perspectives.

Lastly, AI-generated content is often ambiguous, misleading, or biased, requiring human judgement to assess quality. Automated checks like toxicity filters generally can’t fully capture nuances like sarcasm, cultural sensitivities, and ethical concerns.

Practical Techniques

User-Centric Testing. Real users provide feedback on how well AI adapts to changing needs and preferences.
Fact-Checking Panels. Subject matter experts verify AI-generated claims for accuracy and credibility.
Bias and Harm Assessment. Diverse human reviewers assess content for potential ethical issues and unintended harm.
Human Scoring and Annotation. Evaluators rate AI outputs on quality criteria such as coherence, creativity, appropriateness, practicality, among others.

Effectively Scaling Human Evaluation of AIIAs

Over the past 14 months, the team at Test IO has been diligently focused on human evaluation of AI-infused applications for a variety of large enterprise clients. The AI-infused applications under test range from independent chat and voice bots, to code and cloud assistants integrated into software development environments and cloud platforms. So how do you make human evaluation scalable, structured, and reliable? Here are some of the key lessons we’ve learned along the way.

Establish Clear Evaluation Criteria

This involves defining structured rubrics for human reviewers to ensure consistency. Figure 1 shows a sample deliverable including a cross-section of quality criteria.

Figure 1: Quality Criteria Example Showing the Results of Human Evaluation of AIIAs

Leverage Internal and External Communities

Diverse human expertise may be crowdsourced internally from your own pool of people, or externally via user testing communities. As shown in Figure 2, we capture a diverse set of perspectives by using the Test IO crowdsourcing platform to run test cycles using internal employees or external freelancers, or a combination of the two.

Figure 2: Access to Diverse Set of Human Evaluators including Internal Experts and External Freelancers

Combine Human and AI Judges

Automated tools or, for example, another LLM can be used for initial screening, followed by human reviewers for deeper analysis. Not only can this technique be applied to accelerate the evaluation activities, but it also facilitates comparing human versus automated evaluation. The confusion matrix in Figure 3 illustrates the correlation between human ground truths and labels generated by GPT4. Such an artifact can be used to indicate cases where the LLM assigns “irrelevant” to something that the human assigned as highly relevant, and vice-versa.

Figure 3: Confusion Matrix of Human Scores versus LLM Scores

Continuously Incorporate Human Feedback

Crowd-sourced human evaluation of AI-infused applications is applicable to several dimensions of testing. Test cycles can be exploratory, focusing on the early stages of app development or on new features. When issues are discovered, user feedback can be fed back into the model via approaches like reinforcement learning from human feedback. After the given model is updated or re-trained, test cycles can be executed as a form of regression using humans, automated tools, or a combination of both. Figure 4 provides a side-by-side comparison of these two general modes of conducting AIIA testing at scale using crowd sourcing.

Figure 4: Crowd-Sourced Exploratory Testing and Regression Testing of AIIA

Conclusion

For now, AI systems are too complex and dynamic to be tested solely through automation. Human evaluation is indispensable for detecting biases, verifying real-world applicability, and ensuring an ethical and engaging user experience. By integrating structured human oversight and deploying it using a scalable outcome-based model, we can look towards a future where AI systems are not only technically robust, but also aligned with societal values and user expectations.

Author

Tariq King, CEO and Head of Test IO

Tariq King is a recognized thought-leader in software testing, engineering, DevOps, and AI/ML. He is currently the CEO and Head of Test IO, an EPAM company. Tariq has over fifteen years’ professional experience in the software industry, and has formerly held positions including VP of Product-Services, Chief Scientist, Head of Quality, Quality Engineering Director, Software Engineering Manager, and Principal Architect. He holds Ph.D. and M.S. degrees in Computer Science from Florida International University, and a B.S. in Computer Science from Florida Tech. He has published over 40 research articles in peer-reviewed IEEE and ACM journals, conferences, and workshops, and has written book chapters and technical reports for Springer, O’Reilly, Capgemini, Sogeti, IGI Global, and more. Tariq has been an international keynote speaker and trainer at leading software conferences in industry and academia, and serves on multiple conference boards and program committees.

Outside of work, Tariq is an electric car enthusiast who enjoys playing video games and traveling the world with his wife and kids.

EPAM were Exhibitors in EuroSTAR 2025. Join us at EuroSTAR Conference in Oslo 15-18 June 2026.

Understanding Model-Based Testing: Benefits, Challenges, and Use Cases

March 17, 2025 by Aishling Warde

For test engineers seeking a systematic and organized approach to testing, model-based testing offers a powerful toolset. This method involves working with models that guide the testing process.

Besides creating models of tests, you can model, for example, application behavior, application structure, data, and environment. In this article, our core focus will be on testing – so, thinking about what aspects to test and how to do that drives the modeling.

Let’s delve deeper into what model-based testing entails, its benefits, challenges, and scenarios where it is most effective.

What is Model-Based Testing?

Model-based testing is a testing approach that revolves around the use of models. Unlike traditional testing, which involves scrutinizing every intricate detail, model-based testing takes a more general approach. It allows you to concentrate on the core functionalities without being bogged down by all the little details.

Let’s take an example – say that you’re testing an address book application. In this case, you could model the following actions:

• Start the application
• Create a new file
• Add contacts
• Remove contacts
• Save the file
• Open the file
• Quit the application

The idea is not to model the whole application, as a developer would, but rather to get a grasp of the test cases you need to prioritize. This will help in organizing your test cases and in the end your test scripts, which can then be used for automating the test cases.

Benefits of Model-Based Testing

Helps focus on the things that matter
By focusing on high-level abstractions, model-based testing helps you avoid getting lost in the details. This strategic approach allows you to skip unnecessary test cases, optimizing testing efforts and resources.

Ultimately, this leads to higher-quality tests that accurately represent critical functionalities.

Makes communication easier
Models help in finding a common understanding of the requirements and detecting potential misunderstandings. They make it easier to convey testing needs to both internal and external stakeholders.

For example, with models, you could show the management what your test process looks like and why additional resources are needed. Or you could explain to the development team how you’re currently testing and discuss why something is not working as it should.

The visual aid that models offer is often more effective than discussing the problems verbally or looking at abstract test scripts.

Better communication in the early stages of the development process also leads to early detection of bugs – our benefit number 3.

Avoid defects in the early stages of the product
In the traditional development process, the steps of requirements, design, and testing are performed sequentially using a variety of tools. As testing is the final stage, most defects – accumulated throughout the previous stages – are caught quite late in the process. This makes fixing them time-consuming and costly.

Model-based testing is one methodology further enabling so-called shift-left testing. This refers to the shift in the timeline – testing can begin already at the requirement phase.

Models can be shared with project stakeholders, before the implementation, to verify requirements and to identify gaps within the requirements. It might also reveal a problem area if you cannot model something.

As a result, defects are caught and removed earlier, lowering the total cost of development. According to MathWorks, the savings can range from 20 to 60% when compared with traditional testing methods.

Effort reduction in implementation and maintenance

While modeling requires initial effort, it significantly reduces the effort needed for implementation and maintenance.

Model-based testing utilizes the modularization of test cases. In the case of traditional testing, when some element of your application changes, you might have to change every individual test case. With model-based testing, you can use the building blocks, like Legos, and fixing one single block will bring all your test cases up to date.

Also, there are time-saving benefits as you learn to operate in a more organized way. You can detect the highest priority tests – and avoid any redundant work.

Challenges of Model-Based Testing

Mindset transition

Transitioning from a traditional testing process to model-based testing requires a period of adjustment and learning.

Specific skill set required

Not all test engineers may be proficient in abstract modeling. Creating effective models demands skills such as abstract thinking and generalization. To succeed, you need to keep a bird’s eye view of the whole testing process.

Abstraction level challenge

Selecting the right level of abstraction is crucial. Too abstract, and tests may become less useful; too detailed, and the model may be challenging to work with.

However, abstraction inherently involves simplification and can lead to the loss of critical details, potentially overlooking important aspects.

When to Choose Model-Based Testing?

While model-based testing is a powerful tool, it may not be suitable for every scenario. If you’re dealing with a straightforward application, it may be overkill, potentially leading to over-engineering.

However, for complex software systems and teams capable of working at abstract modeling levels, model-based testing proves invaluable.

Conclusion

Model-based testing is a powerful approach that empowers test engineers to focus on testing the critical aspects of the application under test. By leveraging models as high-level abstractions, teams can enhance test quality, reduce effort, and improve communication.

While it requires a shift in mindset and specific skills, the benefits far outweigh the challenges, particularly in complex software environments. As with any testing methodology, the key lies in thoughtful application and adaptation to suit specific project needs.

In the second part of this article we dive into model-based testing best practices and testing tools. Here you will find a real world example on how to achieve model-based testing in Squish.

Author

Sebastian Polzin

Sebastian Polzin, Product Marketing Manager,
Qt Group, Software Quality Solutions

Qt Group were Gold Partners in EuroSTAR 2025. Join us at EuroSTAR Conference in Oslo 15-18 June 2026.

How GenAI is Shaping the Future of Software Testing

March 12, 2025 by Aishling Warde

As software development accelerates, Quality Assurance teams are facing unprecedented pressure to deliver both speed and quality. Despite exponential innovation in software development over the past couple of decades, QA teams still seem to be grappling with the fact that testing is not happening at the same speed development happens. A recent Testsigma webinar surveying 300 QA professionals revealed that 57% consider time management—building, executing, and maintaining tests—their biggest challenge.

The Challenges Holding Us Back

Modern QA teams face several persistent challenges that slow down testing processes:

Slow Testing Cycles
Testing remains bottlenecked by manual-heavy processes and an overemphasis on building perfect automation frameworks. This traditional approach cannot match the speed of modern development cycles.

High Maintenance Overhead
Traditional test automation requires constant script updates, which create significant maintenance costs and technical debt and divert resources from actual testing activities.

Insufficient Test Coverage
Human-defined test cases often fail to anticipate all edge cases, leading to undetected defects. This limitation becomes more pronounced as software complexity increases.

Why Do These Challenges Exist?

A couple of key factors contribute to this misalignment:

Manual testing—Don’t get me wrong—manual testing is not a bad word at all, as some vendors make it out to be. One can never hope to have 100% automation and anyone claiming that is a snake oil salesperson. But, having said that, automation is one of the best things that has happened to the software testing world. The idea is to use automation as a tool to solve everyday tasks at scale.
Automation as the silver bullet – This is not too unrelated to the above point, that automation is being considered the one solution to all of software testing problems. The industry’s push toward automation has had unexpected consequences. Many skilled testers were forced to become programmers or rely heavily on development teams for test automation. This shift often came at the expense of core testing competencies: domain knowledge and user behavior understanding.

The lure of code-driven testing

The world has come long since the first line of code was written. It has truly overhauled our world, and the possibilities it has brought about are endless. While code has revolutionized our world, the heavy focus on code-driven automation has overshadowed essential testing skills. The industry has drifted away from business-driven and user-behavior-driven testing approaches.

The shift to business-driven testing

A promising trend we are observing at Testsigma is the shift to business-driven testing and technology usage as a means to an end. With the advent of codeless technologies and GenAI, it is becoming increasingly easy for software testing teams to automate without having to build a test automation framework that takes months, and without writing code to actually script test cases. This allows testers to focus on their core strengths: domain expertise and user understanding.

GenAI-powered testing

Generative AI, combined with truly codeless test automation, offers new possibilities for rapid testing without coding requirements. Powerful tools like Testsigma Copilot allow testers to:

Focus on core understanding of the business and user, and not necessarily on learning the technology used to build frameworks.
Use prompt engineering to provide business context, to make the testing itself better
Guide AI systems to think from a user’s perspective, and use AI to uncover edge cases that might otherwise get missed

While using GenAI to generate test cases and test scenarios seems to win half the battle, the magic lies in codelessly being able to automate them as well. And that’s where truly codeless test automation platforms like Testsigma help, as one can perform end-to-end test automation at scale without writing a single word of code. With agentic execution, the deal gets sweeter as AI optimizes for the right test cases to be executed, and across the right resources, while self-healing tests to account for any changes that might have been shipped.

Pitfalls in GenAI-driven test automation

It is crucial to remember that, just like any piece of technology, GenAI is a tool. And as the saying goes, a fool with a tool is still a fool. The ones that are able to get the tool to work for them are the ones that win. Mastering the tool itself is not the goal, but how to make it work for the business is.

The other risk in GenAI-driven test automation is that the quality completely relies on the inputs we provide. Again, this is a golden opportunity for software testers, as it forces us to think like users, which is the basic trait of any software tester anyway.

Looking ahead

The future of software testing lies not in writing more automation scripts, but in leveraging AI to handle complexity while humans focus on customer needs and business value. This shift represents an opportunity for software testing teams to:

Return to business-centric and customer-centric testing approaches
Reduce technical barriers to effectively democratize testing and make the industry inclusive
Enable faster, smarter, and more comprehensive quality assurance, to empower software engineering teams to release quality software with confidence!

As software complexity continues to grow, the industry must embrace solutions that streamline testing while keeping quality control in the hands of those who best understand the product and its users.

Author

Narain Muralidharan

Narain Muralidharan is the Director of Product Marketing at Testsigma. Prior to Testsigma, Narain has led various marketing teams at SaaS unicorns like BrowserStack and Freshworks.

Testsigma were Gold Partners in EuroSTAR 2025. Join us at EuroSTAR Conference in Oslo 15-18 June 2026.