Early Bird Savings: 10% off Tickets

EuroSTAR Conference

Europe's Largest Software Testing Conference.

Track Talk, Th1

Automated Testing of Large Language Models: A Live Demo

Anupam Krishnamurthy

09:00 - 09:45 CEST Thursday 18th June

How do you test a large language model’s shape-shifting outputs using automated tests? In this session, I will demonstrate the inner workings of a Retrieval Augmented Generation (RAG) model, and how you can qualify it using automated tests.

Spoiler alert: testing an LLM’s non-deterministic outputs will involve using another LLM to judge this output and score it. We will look at how these automated tests work by using LLM-as-judge.

Once we run the tests, we will prop up the hood and examine the stack traces of the evaluation framework, so that we can debug an unexpected result. We will then subject the RAG model to an indirect injection attack.

The tools used:

LangChain for creating the RAG model

Ragas for performing the assessment

LangSmith for examining the stack traces

Join me in this live demonstration, where we pit one LLM against another, and try to expose a security flaw in the bargain.

What you will learn

  1. A deeper, more intuitive understanding of how LLMs and RAG models work
  2. A method for performing automated RAG assessments - promises and limitations
  3. Using stack traces to understand the assessment and debug unexpected results

Session Details

  • Introductory
  • 45 Minutes
  • Includes 15mins Q&A
  • Testing the reliability, fairness, and safety of AI models
Buy Conference Ticket

Check out this video from Anupam Krishnamurthy to hear more about his talk on Automated testing of Large Language Models: A live demo.

We look forward to welcoming you to EuroSTAR 2026.

Session Speakers

Anupam-Krishnamurthy

Anupam Krishnamurthy

Head of AI Testing – TestSolutions, Germany

Anupam is Head of AI Testing at TestSolutions. In the past, he has served in several roles spanning space science, software development, strategy consulting and process automation. As a manager turned software engineer, he continually chips away at the boundaries that separate the two disciplines. Anupam is currently uncovering testing principles that are applicable to AI augmented software. The non-deterministic nature of AI software, its stochastic behaviour and the need to evaluate subjective outputs present new challenges to the field of software testing. When he is not stuff like this, you’ll find him engrossed in a game of online chess, running long distances with a History podcast, or curled up in a corner with a book.

.

Early Bird Offer

Save 10% on individual tickets until 30th April

The more you bring, the more you save.

Individual

From €2,070

3 Day Ticket*

BOOK TICKET

Groups

From €1,553

Based on 10 Tickets

Book Group

Stay in The Loop

Subscribe to our newsletter and never miss important announcements, updates and special offers from EuroSTAR.

part of the