Programme Launch Offer: Save 20% - Book Now

EuroSTAR Conference

Europe's Largest Software Testing Conference.

Track Talk, Th1

Automated Testing of Large Language Models: A Live Demo

Anupam Krishnamurthy

09:00 - 09:45 CEST Thursday 18th June

How do you test a large language model’s shape-shifting outputs using automated tests? In this session, I will demonstrate the inner workings of a Retrieval Augmented Generation (RAG) model, and how you can qualify it using automated tests.

Spoiler alert: testing an LLM’s non-deterministic outputs will involve using another LLM to judge this output and score it. We will look at how these automated tests work by using LLM-as-judge.

Once we run the tests, we will prop up the hood and examine the stack traces of the evaluation framework, so that we can debug an unexpected result. We will then subject the RAG model to an indirect injection attack.

The tools used:

LangChain for creating the RAG model

Ragas for performing the assessment

LangSmith for examining the stack traces

Join me in this live demonstration, where we pit one LLM against another, and try to expose a security flaw in the bargain.

What you will learn

  1. A deeper, more intuitive understanding of how LLMs and RAG models work
  2. A method for performing automated RAG assessments - promises and limitations
  3. Using stack traces to understand the assessment and debug unexpected results

Session Details

  • Introductory
  • 45 Minutes
  • Includes 15mins Q&A
  • Testing the reliability, fairness, and safety of AI models
Buy Conference Ticket

Check out this video from Anupam Krishnamurthy to hear more about his talk on Automated testing of Large Language Models: A live demo.

We look forward to welcoming you to EuroSTAR 2026.

Session Speakers

Anupam-Krishnamurthy

Anupam Krishnamurthy

Freelance test automation consultant – Anukrit, Germany

Anupam is a freelance test solution architect who has served in several roles in space science, software development, strategy consulting and process automation. As a manager turned software engineer, he continually chips away at the boundaries that separate the two disciplines.

Anupam is committed to continually improve the quality of software by streamlining processes and eliminating waste. When he is not helping teams achieve this, you’ll find him engrossed in a game of online chess, running long distances with a History podcast, or curled up in a corner with a book.

Programme Launch Offer

Save 20% on individual tickets until February 28th!

The more you bring, the more you save.

Individual

From €1,840

3 Day Ticket*

BOOK TICKET

Groups

From €1,380

Based on 10 Tickets

Book Group

Stay in The Loop

Subscribe to our newsletter and never miss important announcements, updates and special offers from EuroSTAR.