EuroSTAR Conference

Europe's Best Software Testing Conference

Track Talk F2

Adversarial Testing for AIs: Get Ready to Fuzz Things Up!

Benjamin Johnson-Ward

09:00-10:00 Friday 14th June

Today’s applications are AI-driven, and AI-built. They challenge established testing concepts like expected results and coverage. How do you “test” an infinitely large set of inputs, with non-deterministic outcomes that can change every time? How do you test applications that rely on external language models that are costly to invoke? What about chains of them, and biases in their training data?

This crash course will teach you to test AI-based systems, pipelines and models. You will learn how to scope your testing, including when to test an AI directly or not. I will then highlight forgotten testing techniques that offer advantages for AI, including fuzzing, metamorphic testing and property-best testing. Next, I will help you upskill in new AI testing techniques and failure types, including adversarial testing using glitch tokens. This will further highlight how you can repurpose current testing methods. I will discuss mocking for testing AI-driven services, profiling training data to optimize quality, and generating synthetic data for testing learning pipelines. I will further offer a “whiter-box” definition of coverage to help you measure your testing of AI models, before calling you to extend Quality Assurance into AI monitoring, engineering and training.

Let’s evolve as AI testers together!

What you will learn

  1. Understand the challenges presented by AI to prevalent testing methods today
  2. Identify where your current testing skills and approaches can be applied to AI - and where forgotten techniques could make a valuable comeback
  3. Learn Quality Engineering techniques that have emerged in the face of AI, and where you should upskill during the ongoing "AI Boom"

Session Details

  • Intermediate
  • 1 Hour
  • Includes 20mins Discussion
  • New & useful skills
Buy Conference Ticket

Session Speaker

Headshot of EuroSTAR 2024 speaker Ben Johnson-Ward

Benjamin Johnson-Ward

Lead Solutions Engineer – Curiosity Software, USA

Before spending the past 10+ years pioneering testing techniques and tools, Benjamin studied economics at the world-leading London School of Economics. He focused on behavioral economics and econometrics, questioning how unconscious biases and economic structures influence behavior. Benjamin has since applied these insights to become a better tester, seeking to understand how unconscious biases can mislead the critical thinking that lies at the heart of testing.

Benjamin has applied this introspective mindset to his work as a tester at both start-ups and multinationals. He has helped shape testing techniques at global banks, retailers, insurance companies, telcos and beyond. He has occupied many of the roles associated with “quality”, including developer, product manager, automation engineer and tester. Benjamin’s work has often gravitated towards model-based testing and test data management. He has most recently focused on applications of Generative AI to testing, both as a product manager and delivery specialist of AI-driven tools.

BACK TO PROGRAMME

Stay in the Loop

We want to ensure you never miss important announcements, updates and special offers from EuroSTAR.