Call for Speakers — Opening Soon

EuroSTAR Conference

Europe's Largest Quality Engineering Conference

Software Testing

Scaling Human Evaluation of AI-Infused Applications

by

It seems like everyone in the quality engineering community is talking about AI. After all, advances generative AI have been rapidly transforming our tools, frameworks, platforms, processes, and ways of working. AI-assisted software lifecycle activities are becoming more widespread and generally accepted.

However, while everyone has been busy applying AI to testing, myself and the team at Test IO have been focused on harnessing the collective intelligence of humans to validate and verify several different types of AI systems. Our approach is currently being utilized to test some of the most sophisticated AI models, assistants, co-pilots and agents at scale.

This article shares our approach to testing what we refer to as AI-infused applications. It describes the grand challenge with testing these types of applications and then discusses the need for human evaluation. After outlining a number of practical techniques, it provides some lessons learned on how to effectively scale the human evaluation of AI/ML.

AI-Infused Applications (AIIA)

Just as with any application, there are many different ways that an AI-based system can be implemented. The particular method used generally depends on the problem being solved, desired capabilities, and any constraints on its development and operation.

Some common approaches to developing AI systems are:

  • Rules-Based AI. Encodes domain expertise into conditional (if-then-else) statements, heuristics, or expert systems.
  • Classical Machine Learning. Training a supervised or unsupervised learning model from scratch using structured datasets and algorithms like random forests, support vector machines, or gradient boosting.
  • Integrating a Pre-Trained Model. Leverages APIs from providers like OpenAI, Google, Anthropic, or Hugging Face to integrate an AI-powered component into an application with minimal development effort.
  • Retrieval Augmented Generation. Combines LLMs with a vector search database to fetch relevant information before generating a response.
  • Fine-Tuning a Pre-Trained Model. Uses transfer learning to adapt a pre-trained model to a specialized dataset to improve the performance of specific tasks.
  • Agent or Multi-Agent Based AI. Builds autonomous agents using an LLM backend, goal-setting mechanisms, and tools for action execution, e.g., APIs, databases, browser automation. These types of systems may include reinforcement learning (RL) or emergent behavior to allow multiple AI agents to interaction, collaborate, or compete in an environment.
  • AI-Orchestrated Workflows. Integrates multiple AI components using workflow orchestration tools such as LangChain, Haystack, or Airflow).

Each of these approaches have trade-offs in terms of accuracy, efficiency, cost, interpretability, among others. However, regardless of the approach used, as long as an application leverages AI or ML models or services as part of their logic, we consider it to be an AI-infused application.

Grand Testing Challenge

The rapid pace of growth in the AI space makes it particularly difficult to keep track of all of the different ways these types of systems can be implemented. However, irrespective of the development method, AI-infused applications present a grand challenge for software testing practitioners primarily due to their highly dynamic nature.

Dynamism

AI-infused applications exhibit different levels of dynamism depending on their purpose and capabilities. For example, there are dynamic aspects of predictive, adaptive, and generative AI systems which make them unpredictable, non-deterministic and, as a result, very difficult to test.

  • Predictive AI analyzes historical data to identify patterns and make forecasts about future events or outcomes. These types of systems evolve with data. In other words, the accuracy of predictions depends on continuously updated data and therefore, as new data arrives, retraining or fine-tuning the model helps improve its forecasts. Some predictive systems like stock trading algorithms process real-time data streams, modifying forecasts as conditions change.
  • Adaptive AI continuously learns from new experiences and environmental changes to modify its behavior and improve performance over time. Unlike traditional models, it evolves without requiring explicit reprogramming. For example, self-learning chatbots will personalize their responses over time. Systems like these are context aware and dynamically adjust based on real-world conditions. Adaptive AI can autonomously tweak its internal models and strategies to improve accuracy and efficiency over extended use.
  • Generative AI creates new content such as text, images, code, and music based on learned patterns from vast datasets. The same user prompts sent to a model can generate different responses. Generative models can refine outputs based on user feedback, style preferences, leading to evolving content quality. Model knowledge can be augmented with external sources via retrieval augmented generation, making the overall system highly flexible.

Adequately testing AI systems may involve a combination of pre- and post release testing, continuous monitoring, automated pipelines, adversarial testing, and human evaluation. These approaches help address quality challenges with AI systems including model drift, bias, fairness, uncertainty, output-variability, explainability, hallucinations, and more.

The Importance of Human Evaluation

While automated testing methods for AI systems help to monitor performance, human evaluation is essential to ensure AI aligns with real-world expectations. Here’s why human evaluation is critical and some techniques that can be applied in practice.

Why It Matters

In classical ML systems, automated accuracy metrics such as F1 scores don’t necessarily capture the real-world impact of predictions. Bias and fairness issues often require domain experts to identify potential harms and when it comes to explainability, although some tools provide insights, human judgement is generally needed to interpret them meaningfully. If an AI-infused application is going to interact directly with users, the system must be assessed for user experience and usability.

Adaptive AI systems can potentially start optimizing on the wrong objectives. For example, a common problem with recommendation systems is that they tend to reinforce their own biases. Here’s how:

  • Recommendation system suggests content based on past user behavior.
  • User engages more with that types of content (e.g., specific movie genres, political articles)
  • System interprets this as a strong preference, leading it to narrow future recommendations to similar content.
  • Over time, diversity in recommendations decreases and users are less likely to be exposed to alternative perspectives.

Lastly, AI-generated content is often ambiguous, misleading, or biased, requiring human judgement to assess quality. Automated checks like toxicity filters generally can’t fully capture nuances like sarcasm, cultural sensitivities, and ethical concerns.

Practical Techniques

  • User-Centric Testing. Real users provide feedback on how well AI adapts to changing needs and preferences.
  • Fact-Checking Panels. Subject matter experts verify AI-generated claims for accuracy and credibility.
  • Bias and Harm Assessment. Diverse human reviewers assess content for potential ethical issues and unintended harm.
  • Human Scoring and Annotation. Evaluators rate AI outputs on quality criteria such as coherence, creativity, appropriateness, practicality, among others.

Effectively Scaling Human Evaluation of AIIAs

Over the past 14 months, the team at Test IO has been diligently focused on human evaluation of AI-infused applications for a variety of large enterprise clients. The AI-infused applications under test range from independent chat and voice bots, to code and cloud assistants integrated into software development environments and cloud platforms. So how do you make human evaluation scalable, structured, and reliable? Here are some of the key lessons we’ve learned along the way.

Establish Clear Evaluation Criteria

This involves defining structured rubrics for human reviewers to ensure consistency. Figure 1 shows a sample deliverable including a cross-section of quality criteria.

Figure 1: Quality Criteria Example Showing the Results of Human Evaluation of AIIAs

Leverage Internal and External Communities

Diverse human expertise may be crowdsourced internally from your own pool of people, or externally via user testing communities. As shown in Figure 2, we capture a diverse set of perspectives by using the Test IO crowdsourcing platform to run test cycles using internal employees or external freelancers, or a combination of the two.

Figure 2: Access to Diverse Set of Human Evaluators including Internal Experts and External Freelancers

Combine Human and AI Judges

Automated tools or, for example, another LLM can be used for initial screening, followed by human reviewers for deeper analysis. Not only can this technique be applied to accelerate the evaluation activities, but it also facilitates comparing human versus automated evaluation. The confusion matrix in Figure 3 illustrates the correlation between human ground truths and labels generated by GPT4. Such an artifact can be used to indicate cases where the LLM assigns “irrelevant” to something that the human assigned as highly relevant, and vice-versa.

Figure 3: Confusion Matrix of Human Scores versus LLM Scores

Continuously Incorporate Human Feedback

Crowd-sourced human evaluation of AI-infused applications is applicable to several dimensions of testing. Test cycles can be exploratory, focusing on the early stages of app development or on new features. When issues are discovered, user feedback can be fed back into the model via approaches like reinforcement learning from human feedback. After the given model is updated or re-trained, test cycles can be executed as a form of regression using humans, automated tools, or a combination of both. Figure 4 provides a side-by-side comparison of these two general modes of conducting AIIA testing at scale using crowd sourcing.

Figure 4: Crowd-Sourced Exploratory Testing and Regression Testing of AIIA

Conclusion

For now, AI systems are too complex and dynamic to be tested solely through automation. Human evaluation is indispensable for detecting biases, verifying real-world applicability, and ensuring an ethical and engaging user experience. By integrating structured human oversight and deploying it using a scalable outcome-based model, we can look towards a future where AI systems are not only technically robust, but also aligned with societal values and user expectations.

Author

Tariq King, CEO and Head of Test IO

Tariq King is a recognized thought-leader in software testing, engineering, DevOps, and AI/ML. He is currently the CEO and Head of Test IO, an EPAM company. Tariq has over fifteen years’ professional experience in the software industry, and has formerly held positions including VP of Product-Services, Chief Scientist, Head of Quality, Quality Engineering Director, Software Engineering Manager, and Principal Architect. He holds Ph.D. and M.S. degrees in Computer Science from Florida International University, and a B.S. in Computer Science from Florida Tech. He has published over 40 research articles in peer-reviewed IEEE and ACM journals, conferences, and workshops, and has written book chapters and technical reports for Springer, O’Reilly, Capgemini, Sogeti, IGI Global, and more. Tariq has been an international keynote speaker and trainer at leading software conferences in industry and academia, and serves on multiple conference boards and program committees.

Outside of work, Tariq is an electric car enthusiast who enjoys playing video games and traveling the world with his wife and kids.



EPAM are Exhibitors in this years’ EuroSTAR Conference EXPO. Join us in Edinburgh 3-6 June 2025.

Understanding Model-Based Testing: Benefits, Challenges, and Use Cases

by

For test engineers seeking a systematic and organized approach to testing, model-based testing offers a powerful toolset. This method involves working with models that guide the testing process.

Besides creating models of tests, you can model, for example, application behavior, application structure, data, and environment. In this article, our core focus will be on testing – so, thinking about what aspects to test and how to do that drives the modeling.

Let’s delve deeper into what model-based testing entails, its benefits, challenges, and scenarios where it is most effective.

What is Model-Based Testing?

Model-based testing is a testing approach that revolves around the use of models. Unlike traditional testing, which involves scrutinizing every intricate detail, model-based testing takes a more general approach. It allows you to concentrate on the core functionalities without being bogged down by all the little details.

Let’s take an example – say that you’re testing an address book application. In this case, you could model the following actions:

• Start the application
• Create a new file
• Add contacts
• Remove contacts
• Save the file
• Open the file
• Quit the application

The idea is not to model the whole application, as a developer would, but rather to get a grasp of the test cases you need to prioritize. This will help in organizing your test cases and in the end your test scripts, which can then be used for automating the test cases.

Benefits of Model-Based Testing

  1. Helps focus on the things that matter
    By focusing on high-level abstractions, model-based testing helps you avoid getting lost in the details. This strategic approach allows you to skip unnecessary test cases, optimizing testing efforts and resources.

Ultimately, this leads to higher-quality tests that accurately represent critical functionalities.

  1. Makes communication easier
    Models help in finding a common understanding of the requirements and detecting potential misunderstandings. They make it easier to convey testing needs to both internal and external stakeholders.

For example, with models, you could show the management what your test process looks like and why additional resources are needed. Or you could explain to the development team how you’re currently testing and discuss why something is not working as it should.

The visual aid that models offer is often more effective than discussing the problems verbally or looking at abstract test scripts.

Better communication in the early stages of the development process also leads to early detection of bugs – our benefit number 3.

  1. Avoid defects in the early stages of the product
    In the traditional development process, the steps of requirements, design, and testing are performed sequentially using a variety of tools. As testing is the final stage, most defects – accumulated throughout the previous stages – are caught quite late in the process. This makes fixing them time-consuming and costly.

Model-based testing is one methodology further enabling so-called shift-left testing. This refers to the shift in the timeline – testing can begin already at the requirement phase.


Models can be shared with project stakeholders, before the implementation, to verify requirements and to identify gaps within the requirements. It might also reveal a problem area if you cannot model something.

As a result, defects are caught and removed earlier, lowering the total cost of development. According to MathWorks, the savings can range from 20 to 60% when compared with traditional testing methods.

  1. Effort reduction in implementation and maintenance

While modeling requires initial effort, it significantly reduces the effort needed for implementation and maintenance.

Model-based testing utilizes the modularization of test cases. In the case of traditional testing, when some element of your application changes, you might have to change every individual test case. With model-based testing, you can use the building blocks, like Legos, and fixing one single block will bring all your test cases up to date.

Also, there are time-saving benefits as you learn to operate in a more organized way. You can detect the highest priority tests – and avoid any redundant work.

Challenges of Model-Based Testing

  1. Mindset transition

Transitioning from a traditional testing process to model-based testing requires a period of adjustment and learning.

  1. Specific skill set required

Not all test engineers may be proficient in abstract modeling. Creating effective models demands skills such as abstract thinking and generalization. To succeed, you need to keep a bird’s eye view of the whole testing process.

  1. Abstraction level challenge

Selecting the right level of abstraction is crucial. Too abstract, and tests may become less useful; too detailed, and the model may be challenging to work with.

However, abstraction inherently involves simplification and can lead to the loss of critical details, potentially overlooking important aspects.

When to Choose Model-Based Testing?

While model-based testing is a powerful tool, it may not be suitable for every scenario. If you’re dealing with a straightforward application, it may be overkill, potentially leading to over-engineering.

However, for complex software systems and teams capable of working at abstract modeling levels, model-based testing proves invaluable.

Conclusion

Model-based testing is a powerful approach that empowers test engineers to focus on testing the critical aspects of the application under test. By leveraging models as high-level abstractions, teams can enhance test quality, reduce effort, and improve communication.

While it requires a shift in mindset and specific skills, the benefits far outweigh the challenges, particularly in complex software environments. As with any testing methodology, the key lies in thoughtful application and adaptation to suit specific project needs.

In the second part of this article we dive into model-based testing best practices and testing tools. Here you will find a real world example on how to achieve model-based testing in Squish.

Author

Sebastian Polzin

Sebastian Polzin, Product Marketing Manager,
Qt Group, Software Quality Solutions



Qt Group are Gold Partners in this years’ EuroSTAR Conference EXPO. Join us in Edinburgh 3-6 June 2025.

How GenAI is Shaping the Future of Software Testing

by

As software development accelerates, Quality Assurance teams are facing unprecedented pressure to deliver both speed and quality. Despite exponential innovation in software development over the past couple of decades, QA teams still seem to be grappling with the fact that testing is not happening at the same speed development happens. A recent Testsigma webinar surveying 300 QA professionals revealed that 57% consider time management—building, executing, and maintaining tests—their biggest challenge.

The Challenges Holding Us Back

Modern QA teams face several persistent challenges that slow down testing processes:


Slow Testing Cycles
Testing remains bottlenecked by manual-heavy processes and an overemphasis on building perfect automation frameworks. This traditional approach cannot match the speed of modern development cycles.


High Maintenance Overhead
Traditional test automation requires constant script updates, which create significant maintenance costs and technical debt and divert resources from actual testing activities.


Insufficient Test Coverage
Human-defined test cases often fail to anticipate all edge cases, leading to undetected defects. This limitation becomes more pronounced as software complexity increases.

Why Do These Challenges Exist?

A couple of key factors contribute to this misalignment:

  • Manual testing—Don’t get me wrong—manual testing is not a bad word at all, as some vendors make it out to be. One can never hope to have 100% automation and anyone claiming that is a snake oil salesperson. But, having said that, automation is one of the best things that has happened to the software testing world. The idea is to use automation as a tool to solve everyday tasks at scale.
  • Automation as the silver bullet – This is not too unrelated to the above point, that automation is being considered the one solution to all of software testing problems. The industry’s push toward automation has had unexpected consequences. Many skilled testers were forced to become programmers or rely heavily on development teams for test automation. This shift often came at the expense of core testing competencies: domain knowledge and user behavior understanding.

The lure of code-driven testing

The world has come long since the first line of code was written. It has truly overhauled our world, and the possibilities it has brought about are endless. While code has revolutionized our world, the heavy focus on code-driven automation has overshadowed essential testing skills. The industry has drifted away from business-driven and user-behavior-driven testing approaches.

The shift to business-driven testing

A promising trend we are observing at Testsigma is the shift to business-driven testing and technology usage as a means to an end. With the advent of codeless technologies and GenAI, it is becoming increasingly easy for software testing teams to automate without having to build a test automation framework that takes months, and without writing code to actually script test cases. This allows testers to focus on their core strengths: domain expertise and user understanding.

GenAI-powered testing

Generative AI, combined with truly codeless test automation, offers new possibilities for rapid testing without coding requirements. Powerful tools like Testsigma Copilot allow testers to:

  • Focus on core understanding of the business and user, and not necessarily on learning the technology used to build frameworks.
  • Use prompt engineering to provide business context, to make the testing itself better
  • Guide AI systems to think from a user’s perspective, and use AI to uncover edge cases that might otherwise get missed

While using GenAI to generate test cases and test scenarios seems to win half the battle, the magic lies in codelessly being able to automate them as well. And that’s where truly codeless test automation platforms like Testsigma help, as one can perform end-to-end test automation at scale without writing a single word of code. With agentic execution, the deal gets sweeter as AI optimizes for the right test cases to be executed, and across the right resources, while self-healing tests to account for any changes that might have been shipped.

Pitfalls in GenAI-driven test automation

It is crucial to remember that, just like any piece of technology, GenAI is a tool. And as the saying goes, a fool with a tool is still a fool. The ones that are able to get the tool to work for them are the ones that win. Mastering the tool itself is not the goal, but how to make it work for the business is.

The other risk in GenAI-driven test automation is that the quality completely relies on the inputs we provide. Again, this is a golden opportunity for software testers, as it forces us to think like users, which is the basic trait of any software tester anyway.

Looking ahead

The future of software testing lies not in writing more automation scripts, but in leveraging AI to handle complexity while humans focus on customer needs and business value. This shift represents an opportunity for software testing teams to:

  • Return to business-centric and customer-centric testing approaches
  • Reduce technical barriers to effectively democratize testing and make the industry inclusive
  • Enable faster, smarter, and more comprehensive quality assurance, to empower software engineering teams to release quality software with confidence!

As software complexity continues to grow, the industry must embrace solutions that streamline testing while keeping quality control in the hands of those who best understand the product and its users.

Author

Narain Muralidharan

Narain Muralidharan is the Director of Product Marketing at Testsigma. Prior to Testsigma, Narain has led various marketing teams at SaaS unicorns like BrowserStack and Freshworks.

Testsigma are Gold Partners in this years’ EuroSTAR Conference EXPO. Join us in Edinburgh 3-6 June 2025.

How to Measure the Value of Software Testing EXPO Trade Show Participation

by

Participating in trade shows like the EuroSTAR Software Testing EXPO can be a significant investment of time, money, and resources. However, the potential benefits, including increased brand visibility, lead generation, and networking opportunities, can be substantial. To ensure that your participation is worthwhile, it is crucial to measure the value derived from the event. This blog post outlines Clare’s recommended key metrics and strategies to help you evaluate the return on investment (ROI) from participating in the EuroSTAR Software Testing EXPO.

Set Clear Objectives

Before delving into metrics, it’s essential to define your goals for participating in the trade show. These goals should align with your overall marketing strategy. Common objectives include:

  • Lead Generation: Capturing contact information of potential clients.
  • Brand Awareness: Increasing visibility and recognition of your brand.
  • Networking: Building relationships with industry peers and potential partners.

Setting clear objectives when participating in an EXPO helps us ensure we are enabling you to focus efforts, measure success, allocate resources effectively, and enhance overall strategic alignment with business goals. Clear objectives are helpful when we are helping you pick an EXPO package that best suits your business goals.

Pre-Event Preparations

To measure the value effectively, start tracking metrics before the event:

  • Budget Allocation: Document all expenses related to the event, including booth costs, travel, accommodation, marketing materials, and promotional items.
  • Marketing Reach: Assess your pre-event marketing efforts, such as email campaigns, social media promotions, and blog posts. Depending on which EXPO package you have opted for there may be pre-conference marketing activations that will also help boost your brands attendance and visibility.

Lead Generation

One of the primary reasons for attending the EXPO is to generate leads. Collecting leads is a very important part of exhibiting as well ad forging connections and networking. Key metrics to track include:

  • Number of Leads Collected: Count the total number of leads gathered during the event.
  • Lead Quality: Evaluate the quality of leads based on criteria such as job titles, company size, and level of interest.
  • Lead Conversion Rate: Track how many leads convert into actual sales or follow-up meetings, this is dependant on the actions taken post event.

Brand Awareness & Engagement

Increasing brand awareness and engagement is another crucial objective. Things to consider:

  • Booth Traffic: Monitor the volume of visitors to your booth each day, try different initiatives to encourage footfall and get involved in the EuroSTAR EXPO Passport around the EXPO. Some other ideas include running a competition or have a testing challenge.
  • Social Media Engagement: Track mentions, shares, likes, and comments on your social media posts related to the event. Always be sure to share your participation in the event. This helps build awareness and visibility for your brand
  • Media Coverage: Monitor any press coverage or mentions in industry publications resulting from your participation.
  • Swag: Keep your brand at the forefront of attendee’s minds with a cool piece of conferences swag for them to take home. Our attendees love conference swag.

Market Research & Networking

Understanding the software testing industry trends and building relationships can be invaluable. Things to be conscious of:

  • Competitive Analysis: Mingle with other exhibitors in the EXPO. Having conversations helps gain insights about other companies products and services, challenges, successes, pricing, and strategies.
  • Partnership Opportunities: Count the number of potential partnership discussions initiated.
  • Feedback and Insights: Collect feedback from conversations with attendees and industry experts to identify trends and areas for improvement.

Post-Event Follow-Up

Effective follow-up is critical to maximising the value of EXPO participation:

  • Timely Follow-Up: Ensure that leads are contacted promptly after the event.
  • Nurturing Campaigns: Implement nurturing campaigns to keep leads engaged and move them through the sales funnel.
  • Feedback Surveys: Conduct surveys to gather feedback from attendees and improve future participation.

Conclusion

Measuring the value of your participation in the EuroSTAR Software Testing EXPO requires a systematic approach and a focus on relevant metrics. By setting clear objectives from the beginning, tracking key metrics, and continuously improving your strategy, you can ensure that your investment in the EuroSTAR Software Testing EXPO will deliver substantial returns. With diligent measurement and follow-up, you can leverage EXPO participation to boost your brand, generate quality leads, and drive business growth.

To find out how you can achieve your marketing goals and more at a EuroSTAR Conferences EXPO, speak with Clare.

Clare Burke

EXPO Team, EuroSTAR Conferences

With years of experience and a passion for all things EuroSTAR, Clare has been a driving force behind the success of our EXPO. She’s the wizard behind the EXPO scenes, connecting with exhibitors, soaking up the latest trends, and forging relationships that make the EuroSTAR EXPO a vibrant hub of knowledge and innovation. 

t: +353 91 416 001 
e: clare@eurostarconferences.com 

Innovating quality: strategic approaches to complex software testing

by

As the complexity of modern solutions increases, not only does the challenge for software developers grow, but so do the expectations for software testers. Although values such as ‘lines of code’ are not universally meaningful metrics, the following graph shows well where the journey is heading.

In the early days of computer technology, programs were relatively simple and straightforward. However, as technology advanced and software applications became more powerful, the size of code bases increased significantly. This is nothing wrong, of course, but it does provide new challenges.

Increased complexity – more bugs

As software has evolved, so has its complexity. Software systems now encompass complex architectures, sophisticated algorithms, and diverse integrations with various platforms and technologies. The demand for advanced functionalities, seamless user experiences, and robust security measures has led to the development of complex software applications. Managing this complexity requires careful design, modularization, and the adoption of architectural patterns to ensure maintainability and scalability.

While software has become more sophisticated and powerful, the presence of bugs remains a challenge. As software grows in size and complexity, the potential for defects and errors also increases. The sheer volume of code, coupled with interdependencies between modules and integration with external components, introduces a higher likelihood of bugs.

The need for faster product delivery can impact product quality

Companies are increasingly under fire to deliver software with new functionalities and improvements to stay ahead of the competition. This puts pressure on all involved roles to deploy new code rapidly, which often results in reduced testing efforts and even more bugs in production.

In today’s competitive business environment, a company’s reputation is also closely tied to the perceived and experienced reliability of its services. Even the smallest of bugs can have a significant negative impact on a company. In conclusion, testing is a crucial discipline to identify bugs before they are discovered by the end-users to secure the company’s reputation.

The crucial role of testers in product quality assurance

In this game, testers play a crucial role in safeguarding product quality by ensuring that the product meets the required standards in terms of functionality, reliability, usability, and more. This can’t be achieved just by clicking around in an application and reporting some “strange behaviors”. Testing should be a comprehensible activity based on specification, proven methods, and an overall understanding of the business context.

To ensure product functionality and quality, you need skilled people with a quality-focused mindset, who understand business processes as well as the implemented technologies and solutions. This work cannot be a “last minute” task that can be skipped in case of tight deadlines or done by any person “just available”.

Shift testing to the left and involve testers from the beginning

So, how is it possible to implement quality assurance in such a way that it does not delay the delivery date and still has an actual effect?

This requires involving experienced testers who have the capability to analyze the use cases and IT solution design. Even more crucially, they should be involved from the very beginning of the process. By challenging business ideas and identifying missed elements or flows early on and even without code, testers can help avoid wasting unnecessary resources on functionalities that may need to be changed later. This calls for close collaboration between business, testers, and developers as one team.

Starting testing early in the software development life cycle (SDLC) is known as the “Shift left” approach, and it comes from the idea of shifting test activities to the “left” in the development process timeline to ensure problems are detected as early as possible. Involving testers right away, starting with the requirements gathering stage, brings numerous benefits, such as early defect detection, faster feedback to developers, better collaboration between teams, and reduced business risk.

Efficient testing without excessive costs

In the realm of rapid code deployment, financial investments alone won’t solve quality issues. In addition to “Shift left” there are different methods that can ensure successful testing efforts without increasing the testing budget.

Testing isn’t just about speed; it’s a strategic blend of prioritization and disciplined collaboration. At Sixsentix, we advocate a disciplined testing approach centered on business-facing testing and business risk coverage.

With our risk-based methodology, we can clearly identify the processes and features with the greatest business impact, based on the frequency of use and potential damage that can occur as a result of failure. We perform a thorough analysis of the business, accurately determine specific business risks thanks to the combination of expertise and domain knowledge, and then prioritize testing of sensitive components.

This way, we make sure that key functionalities are swiftly checked and validated, and that the most pressing business risks are ruled out. Sixsentix combines this approach with the “Shift left”, which allows continuous testing and deployment after every change. This means that the latest updates are always released as soon as possible, without critical failures that could damage your reputation and business long-term.

Enhancing software quality through team dynamics

It’s also crucial to have a cohesive team composition, backed by a time-tested method. Our testing teams have been using our well-refined SWAT methodology, which enables them to use a systemic approach to testing.

Simultaneously, we abide by the “Tres Amigos” principle. It highlights the participation of three key roles: the product owner, the developer, and the tester, and emphasizes that these three roles are, in fact, one team, pursuing the same goals and priorities, which significantly boost the delivery of high-quality software.

Finally, in order to orchestrate and automate testing activities efficiently, we also designate a test architect for each project to define the overall testing strategy and framework and ensure they resonate with business objectives. This orchestrated effort significantly enhances the testing process, elevating software quality and ensuring a successful, reliable product.

Moving forward

Transforming quality assurance involves continuous improvement, robust collaboration, and the embrace of automation, propelling organizations towards a mature QA level where testing adapts to evolving requirements and aligns with changing organizational needs.

Let’s embrace efficient testing strategies and disciplined collaboration, striving for a future where software isn’t just fast and functional but also robust, reliable, and aligned with both user needs and business goals.

Author

Sixsentix

Sixsentix is a leading provider of Software Testing Services, QA Visual Analytics and Reporting, helping enterprises to accelerate their Software Delivery. Our unique risk-based Testing and QACube ALM Reporting and Dashboards, provide business with unprecedented quality and transparency across Software Delivery projects for faster time-to-market. Sixsentix customers include the largest banks, financial services, insurance, telecom providers and others. Sixsentix Onsite, and Nearshore (SWAT) services deliver optimized testing outcomes at significantly lower costs and help customers with scalability to keep pace with digitalization.

Sixsentix is an EXPO Exhibitor at EuroSTAR 2024, join us in Stockholm.

Autoethnographic research made me look at testing from a new angle

by

Explore how autoethnographic research techniques overlap with testing behaviors and download a free template to try it yourself.

One of my favorite things about being a tester is the way testers find inspiration and techniques from outside of the testing world and integrate them into their work. In my graduate work I’ve been discovering the many ways that traditional research techniques overlap with testing behaviors, my favorite of which is autoethnographic research. 

What in the world is autoethnographic research you ask? Let’s start by looking at ethnography, the research methodology this technique grew out of. Ethnography or ethnographic research is a research method typically used in the social sciences in which the researcher embeds and immerses themselves in a group, community, or culture. In ethnography we acknowledge that researchers will always have a personal slant or bias and instead of attempting to remove bias, instead seeks to learn and empathize with the subjects of the study. Some of the ways we document our findings using this method include journaling, field notes, and narratives of our experiences and observations.

Autoethnography takes ethnographic research one step further and uses the researcher themselves as a subject in the study. Often used in the social sciences, the researcher interrogates their own feelings and experiences in a group, community, or culture. This technique is often used when a researcher is studying their own culture or community, or when they are conducting research on a study of only themselves. Autoethnographic research also uses journaling, field notes, and narratives to document observations during the study.

A couple of real life examples of this technique used in action are:

  • In James Osben’s A day in the life of a NHS nurse in 21st Century Britain: An auto-ethnography: James wrote a narrative of his day as it happened. He then used critical analysis to find the overarching themes of feelings, tasks, and behaviors in the narrative of his shift. Osben’s aim in his research was to use “reflective practice.” Reflective practice is one of the ways a researcher reflects on, contextualizes, and reframes their experiences for their research.
  • Jay Johnson Theil explored her place in academia using ethnographic methods in her research Working-class women in academic spaces: finding our muchness: Theil used photography, reflective journaling, movie quotes, and storytelling as research methods, then applied critical analysis to better understand how she really feels about her role in academia. 

Autoethnographic research works a lot like exploratory testing

If I’ve explained this well enough you may already see what I saw when first learning about this technique — autoethnographic research sounds like exploratory testing! I’ve long advocated for paying close attention to your thoughts and feelings as you work your way through an exploratory testing session. With this research method, you can take your exploratory testing even further. Here are a few ways to apply this method to your testing. 

Journal your feelings while testing. As you work your way through whatever you’re testing, focus on how you feel and what you think. Do you feel frustrated, delighted, or confused? Whatever feelings the application elicit from you, write them down as you feel them. You may also consider other types of journaling using methods like color coded or face sentiment trackers.

Use voice narration. Narrate your thoughts and feelings to a voice app or Zoom session. To make sharing the narrated session easier for your team, you can use a transcription app or speech to text.

Reflect at the end of testing sessions. If you’re more interested in overall sentiment as opposed to thoughts and feelings in the moment, consider writing or narrating a reflection of thoughts and feelings from the testing session. 

Debrief with your peers. Many teams hold debriefing sessions on a regular basis after exploratory testing. These sessions are the perfect time to share your thoughts and feelings documented in the session.

I will be the first to admit that I can’t remember what I did yesterday. Over time, I tend to forget my feelings towards the application I’m testing, which can hinder quality because my brain isn’t attune to looking for trends. Starting to use these autoethnographic techniques on an individual level will help you uncover trends in your reactions, thoughts, and feelings related to the application while you test. 

You’ll have an even more impactful and beneficial experience if you use these techniques with your team and track over time. Some ways to do so are:

  • Bug summary journal: Keeping a log of bugs you’ve found and experienced in your application along with sentiment tracking towards each bug helps demonstrate the impact on quality that you feel each bug has. This will also help you see if there are periods when you feel bugs are related to sloppy code or a lack of understanding of the problem the team is trying to solve. 
  • Daily journaling: Tracking your thoughts and feelings about the application on a daily basis will give you a timeline of your sentiment toward the application. This ongoing log gives you an opportunity to identify trends and track confidence in quality in your application over time.
  • Daily reflections: Similar to reflections at the end of a testing session, a daily reflection gives you a summary of the day. Over time, these reflections will expose trends and when compared with peers, may expose shared thoughts and feelings about the application that haven’t been identified previously. 
  • Daily sentiment tracking: A lighter lift than some of the other options that can still expose trends. Each team member tracks their sentiment towards the application each day using either faces, color codes, or other tracking methods. This simplified tracking can help demonstrate changes in confidence in quality over time.

Integrating these techniques is a great way to help your team shift from testing to quality advocacy and demonstrate your ability to identify the quality indicators that have the biggest impact on the overall health and usability of your application.

Give these techniques a try with a free downloadable template from our Github resources.

Author

Jenna Charlton, QA and Developer Advocate

Jenna is a QA and Developer Advocate. They’ve spoken at a number of dev and test conferences and are passionate about testing, agile, teams, and developing the next generation of testers

Qase is an exhibitor at EuroSTAR 2024, join us in Stockholm.

Empowering Functional Manual Testers

by

Let’s be honest. The dream of 100 percent test automation often turns out to be a nightmare. After all, it’s not just about writing a few scripts. Successful test automation needs to be well thought out, requires a test architecture and, above all, time. If you let the reins slip for even a few iterations, technical debt will creep in. In the worst case, the test cases become flaky. 

Less spectacular, but just as painful in the long run, is gentle erosion. While the system to be tested is constantly evolving, the automated scripts lag behind. They may still run successfully, but over time they become less and less meaningful.

“Not true!”, you are probably thinking. “We do test-driven development.” That’s laudable, but probably only part of the truth. Because you certainly have them too – the functional manual testers who cover the upper part of the test pyramid and look at the system as a whole, perhaps even in interaction with other applications. 

The Importance of Functional Testing

Functional testing is important. Of course, as much as possible should be checked during unit testing, but to be honest: the nasty bugs are hidden between the components. An example:

An app requires 2-factor authentication for login. Therefore, an 8-digit code consisting of letters and numbers is sent by email. Unfortunately, the app only allows numbers to be entered.

This example is not made up and illustrates the difference between unit and system testing. Each component complied with its specification and was tested successfully. Nevertheless, the overall system is unusable because the error was not in the code, but in the specification. Functional testers find such errors and that is what makes them so valuable.

Manual testing has never been easy, but today, in the age of Agile, it’s a pure race against time. If they don’t want to be left behind, they need to understand the intricacies of automated functional testing. Because test automation relieves them of the repetitive regression tests and allows them to focus on the important things, such as new features.

Fundamental Principles of Effective Functional Test Design

In fact, there are several common principles that are important in both manual testing and test automation:

  • Clarity : Clarity is key. Structured, easy-to-grasp tests improve comprehension and minimize ambiguity, benefiting all stakeholders. Visual diagrams have long helped developers simplify complex problems. Testing can benefit from similar visualization, as functional tests mirror the intricate nature of systems under test. Visual test design definitely improves clarity, making it easier for test automation engineers to understand the business aspects of functional testing.
  • Modularity : By breaking down complex scenarios into manageable test cases, manual testers can lay the groundwork for seamless automation, ensuring that each test remains a valuable asset throughout the software development lifecycle.
  • Maintainability : Functional tests will continue to evolve, as will the associated test scripts. Some changes affect the technical level, others the functional level. Keyword-driven testing is a proven method of separating these two levels. Manual testers can thus contribute to the maintenance of automated tests without having to program themselves.

The goal of an effective, functional test design must therefore be to develop tests that are easy to maintain and update.

Transitioning from Manual to Automated Execution

Automation is often viewed as process optimization, as it takes over error-prone, repetitive tasks. Testers can use the time gained to focus on those tasks that require human judgment. Executing existing tests repeatedly following the accelerating rhythm of testing cycles is certainly not a mentally demanding task. The dumb writing down of test procedures is not a great intellectual accomplishment either. The real value lies in the preliminary considerations: What situations could occur? How should the system behave in this case? What can I do to push the system beyond its limits? Automation is therefore also a helpful and welcome support when creating test cases.

Studies[1] show that the combination of intellectual performance in test design and automated test execution not only increases test speed, but also the coverage of requirements and code. In general, the reliability of tests is increased when automated scripts support manual testers, especially in lengthy tests. This can be seen very clearly in load and performance tests, which are unthinkable without automation.

An intelligent automation strategy effectively balances human expertise with automated tasks.

A Practical Case Study – Yest Augmented by Maveryx

In this example, we show the successful marriage of two concepts represented by Yest and Maveryx.

Yest is a visual test design tool that implements a modern form of model-based testing (MBT) and test generation. Yest offers a whole range of functions to enhance the creative work of test design and speed up the painstaking task of writing test cases. Yest itself is agnostic, meaning that the generated scenarios may be used both for manual and/or automated test execution.

Maveryx is an automated software testing tool that provides functional UI, regression, data-driven and codeless testing capabilities for a wide range of desktop and Web technologies. Its innovative and intelligent technology inspects the application’s UI at runtime as a senior tester does. With Maveryx, there is no need for code instrumentation, GUI capture, maps and object repositories.

From Visual Test Design for Manual Tests…

With Yest Augmented by Maveryx, the functional manual testers concentrate on creating graphical workflows and define actions including the expected results for the various cases to be tested. Yest then generates test scenarios from these workflows and the stored information, which can be executed directly manually.

…to Automated On-The-Fly Test Execution

But can they also be automated and, if so, what results can we expect? This is where Maveryx comes into play. All you have to do is to provide a detailed instruction of manual steps in Yest. Yest Augmented by Maveryx recognizes these instructions for manual execution (for example “Click on Submit”) and executes them automatically.

If a step is not interpretable or fails, Yest Augmented by Maveryx stops and waits for manual input. You may then abort the test or do the necessary steps manually and continue. The execution results are reported in Yest Augmented by Maveryx or in the test management you read the test cases from.

Use Cases

Yest Augmented by Maveryx serves in various situations:

  1. Hardening the test procedure : Manual tester may execute their tests without having to code a single line of code. This provides them with rapid feedback on the quality of their tests. No more tedious analyses and hot disputes about whose fault it was.
  2. Executing manual tests directly from Xray or MS Excel : During official test runs, manual testers can call upon Yest Augmented by Maveryx to do part of the work. In fact, it is possible to import tests from Xray or MS Excel and execute them even without having used Yest Augmented by Maveryx for test design.

As functional manual testers embark on their automation journey, they will gain valuable insights into creating robust, maintainable test suites that stand the test of time.

Conclusion

This blog aims to guide manual functional testers through the complicated process of developing functional tests and harnessing the transformative power of automation. In an age where software development demands agility and speed, understanding the intricacies of automated functional testing is critical for testers looking to optimize their workflows.

Clarity, modularity and maintainability are key success factors for a successful transition from manual to automated test execution. Visual test design and model-based test case generation pave the way to structured tests and complete coverage. With the right tool support, it is possible to carry out these tests automatically without having to write a single line of code. Yest Augmented by Maveryx provides this support. Contact us to learn more at info@maveryx.com or contact@smartesting.com

[1] Khankhoje, Rohit. (2023). Revealing the Foundations: The Strategic Influence of Test Design in Automation. International Journal of Computer Science and Information Technology. 15. 10.5121/ijcsit.2023.15604.

Authors


Alfonso Nocella, Co-founder and Sr. Software Engineer at Maveryx 

Alfonso led the design and development of some core components of the Maveryx automated testing tool. He collaborated in some astrophysics IT research projects with the University of Napoli Federico II and the Italian national astrophysics research institute (INAF). Over the decades, Alfonso worked on many industrial and research projects in different business fields and partnerships. Also, he was a speaker at several conferences and universities.

Today, Alfonso supports critical QA projects of some Maveryx customers in the defense and public health fields. Besides, he is a test automation trainer, and he takes care of the communication and the technical marketing of Maveryx.


Anne Kramer, Trainer and Global CSM at Smartesting

Anne Kramer first came into contact with model-based test design in 2006. Since then she has been burning for the topic. Among other things, she was co-author of the “ISTQB FL Model-Based Tester” curriculum and lead author of the English-language textbook “Model-Based Testing Essentials” (Wiley, 2016).

After many years of working as a process consultant, project manager and trainer, Anne joined the French tool manufacturer Smartesting in April 2022. Today, she is fully dedicated to using models for testing purposes. This includes visual test design and, more recently, generative AI.

Maveryx is an Exhibitor at EuroSTAR 2024, join us in Stockholm.

How to Build a More Valuable End-to-End Testing Strategy

by

End-to-end testing is one of the most effective ways software teams can understand the customer experience. Unlike unit or component testing, which focus on individual pieces of the application, E2E tests seek to understand product quality as an integrated journey. In many ways, end-to-end testing exemplifies the expanding role of software testing in a DevOps world: the crucial connection between how software is built and how it is used. When development teams understand how their changes will impact their end users, they’re better able to deliver value to those customers. When automated effectively, end-to-end testing provides this connection quickly enough to support continuous delivery.

But the shift to digital-first experiences means that end-to-end testing needs to evolve and expand, running contrary to established testing best practices. Even a simple user journey, such as the one outlined below, likely involves multiple third-party APIs and services as well as email touchpoints and personalized offers or recommendations. Development teams must build seamless user experiences that make a complex customer journey feel simple. Expanding the definition of end-to-end testing ensures they can do so successfully. But with traditional testing frameworks, complicated automated testing meant a high risk for broken tests, extra maintenance, and inaccurate results, which ultimately slowed down development pipelines. Quality teams instead opted for shorter, simpler end-to-end tests that were less likely to break as the product evolved. But the holistic view provided by true end-to-end testing is extremely valuable – if quality teams have the tools to manage them.

New Customer Journeys Demand Broader End-to-End Testing

the testing pyramid and an example end-to-end test for an ecommerce website

The image above includes an example of an end-to-end test for an ecommerce website. Despite this being a fairly simple – and common – transaction, an automated test needs to cover a marketing email, a coupon code, and an invoice email with a PDF attachment. But the story doesn’t end there: it’s extremely likely that the checkout test step includes an API for a payment service like Square or Afterpay. It’s also likely that coupon codes are personalized for customers, given that loyalty programs with customized rewards are proven to increase consumer spending.

Skip these steps, and there is a real risk to revenue. If a marketing email fails to accurately show a customer’s coupon code, conversion rates will suffer, impacting sales and potentially churning previously loyal customers. Managing this type of comprehensive test is essential for supporting quality customer experiences.

The Challenges of Comprehensive End-to-End Testing

Though the above end-to-end test is critical for understanding the user experience and how each change will impact it, such tests pose several challenges for developers and software testers. First, maintaining such an extensive automated test with scripted test automation frameworks is likely to consume a significant amount of a testing team’s time and effort, which has a serious impact on an organization’s ability to accelerate product velocity. Since additional test steps increase the risk of a test breaking, most quality teams avoid creating longer end-to-end tests in order to reduce the burden of test maintenance. But what they gain in reliability, they lose in test coverage.

Second, comprehensive automated tests often require longer investigations into failures. Combing through a long list of test steps to identify the specific cause of a test failure can take valuable hours, a luxury development teams don’t have as delivery cycles shorten. Considering that 44% of developers say that investigating failed tests is a significant pain point, quality teams must have effective strategies in place to triage comprehensive end-to-end tests when necessary.

Maintaining More Complex End-to-End Tests

An end-to-end test covering email, API, and non-functional test steps is highly susceptible to any product changes, but advances in AI and machine learning have reduced the amount of time and effort needed to maintain automated tests, making it possible for quality teams to manage comprehensive end-to-end tests. Using unique identifying elements across an application’s UI, including shadow DOM components, intelligent test automation solutions can detect product changes and update end-to-end tests accordingly.

Automating end-to-end test maintenance not only ensures that test maintenance is less labor-intensive, but also allows more team members to contribute. For example, manual testers can more easily collaborate on E2E tests that contain integrated API tests, ensuring that comprehensive end-to-end tests capture the full user journey and accurately assess quality.

Reporting on End-to-End Testing

Even when end-to-end tests are maintained, identifying the root cause of an error can be time-consuming, causing delays and disruptions in the later stages of the SDLC. Rapid results that support fast bug resolution is critical for delivering exceptional user experiences at the speed of DevOps.

Advances in cloud-based testing and the availability of SaaS test automation tools are making it easier to scale and maintain comprehensive end-to-end testing strategies. Cloud-based runs give in-depth insights that support continuous improvement, and can be run on a schedule or as part of a CI/CD pipeline. Flexible execution options make it possible to routinely and reliably run comprehensive end-to-end tests without slowing development. But perhaps even more importantly, integrating end-to-end test automation into existing development workflows allows developers to quickly act on end-to-end test results.

Building workflows that surface comprehensive end-to-end test results in a digestible way supercharges their value. Sharing test results as Jira tickets, complete with screenshots of the point of failure, DOM snapshots, and performance logs, is ideal for triaging comprehensive end-to-end tests since developers can easily identify what test step caused the failure. The time from failure to fix becomes much shorter, making comprehensive end-to-end tests highly actionable.

The Future of End-to-End Testing

comprehensive end-to-end tests are often considered too time-consuming to provide real value to development teams. But their ability to ensure quality of the perspective of the customer is invaluable, even essential, in a time where every business is competing on their digital customer experience. Overcoming test maintenance, execution, and investigation obstacles to comprehensive end-to-end tests gives development organizations a powerful tool for understanding how changes will impact their users. And with the right test automation solution, end-to-end testing becomes an adaptable process that can continuously evolve to match real customer needs. A few examples include automated accessibility checks, integrated API tests, shadow DOM components, and cross browser testing. No matter what your customers need, comprehensive end-to-end tests will help your team deliver exceptional user experiences. 

Author


Bridget Hughes., Content Marketing Manager at mabl

Bridget is the Content Marketing Manager at mabl, the unified test automation platform for delivering modern software quality. She’s dedicated to helping quality teams expand testing and improve product quality through educational blogs, articles, and the occasional software testing meme.

Mabl is an Exhibitor at EuroSTAR 2024, join us in Stockholm.