Last chance for tickets - Groups save up to 25%

EuroSTAR Conference

Europe's Largest Software Testing Conference.

Track Talk, Th1

Automated Testing of Large Language Models: A Live Demo

Anupam Krishnamurthy

09:00 - 09:45 CEST Thursday 18th June

How do you test a large language model’s shape-shifting outputs using automated tests?

LLMs have boundless input possibilities, and the same input can often produce different outputs. Further, some requirements require subjective evaluation. E.g. The chatbot should be polite, provide concise answers, and address our customers with an informal friendly tone. All of these pose challenges to automated testing.

The technique LLM-as-Judge has emerged as a promising solution. This technique involves prompting another LLM to evaluate the outputs of the LLM-under-test. However, these LLM judges need to be aligned with human judges, who are often domain experts. And this can be easier said than done.

In my talk, I walk you through a structured approach for aligning LLM judges with human experts with a live demo. Our system-under-test is a RAG based system, which generates responses to a list of questions. We then critically evaluate these responses with a live evaluation for a few questions. We then compare our collective judgement with that of an LLM-judge. How far does the machine’s evaluation align with our evaluation?

What you will learn

  1. A structured approach for evaluating LLM based applications
  2. An understanding of RAG, and the LLM-as-judge technique
  3. Understanding how a human-judge baseline can be used to pick a suitable LLM-judge model
  4. An appreciation for the challenges involved with using LLM-judges

Session Details

  • Introductory
  • 45 Minutes
  • Includes 15mins Q&A
  • Testing the reliability, fairness, and safety of AI models
Buy Conference Ticket

Check out this video from Anupam Krishnamurthy to hear more about his talk on Automated testing of Large Language Models: A live demo.

We look forward to welcoming you to EuroSTAR 2026.

Session Speakers

Anupam-Krishnamurthy

Anupam Krishnamurthy

Head of AI Testing – TestSolutions, Germany

Anupam is Head of AI Testing at TestSolutions. In the past, he has served in several roles spanning space science, software development, strategy consulting and process automation. As a manager turned software engineer, he continually chips away at the boundaries that separate the two disciplines. Anupam is currently uncovering testing principles that are applicable to AI augmented software. The non-deterministic nature of AI software, its stochastic behaviour and the need to evaluate subjective outputs present new challenges to the field of software testing. When he is not stuff like this, you’ll find him engrossed in a game of online chess, running long distances with a History podcast, or curled up in a corner with a book.

.

Last Chance for Tickets

Don’t miss out – book tickets now.

Teams save up to 25%.

Individual

From €895

*1 Day Ticket

BOOK TICKET

Groups

From €1,268

Based on 10 Tickets

Book Group

Stay in The Loop

Subscribe to our newsletter and never miss important announcements, updates and special offers from EuroSTAR.

part of the