Track Talk W4

Testing Metrics: The Good, the Bad and the Ugly

Ezanne Grobler

10:30-11:30 Wednesday 12th June

Peter Drucker’s famous assertion, ‘What gets measured gets managed’, and Lord Kelvin’s wisdom, ‘If you can’t measure it, you can’t improve it’, highlight the importance of metrics and measurements. Yet, these aphorisms often overly prioritize numerical values, overlooking the narratives that numbers can convey. This prompts us to ponder on ‘what are we doing here?’ and ‘what is the true essence of quality?’

Gerald Weinberg contended that ‘Quality is value to some person(s)’, and Bach & Bolton stressed understanding ‘…who matters’. In the realm of large-scale transformation projects, the quest for meaningful testing metrics remains elusive. However, by sharing our experiences, I aim to illuminate the valuable lessons drawn from a spectrum of testing metrics, including the good, the bad, and the ugly.

The ‘ugly’ side reveals a transformation program with multiple development vendors, an outsourced project office, an in-house UAT team, and our role as the testing vendor responsible for both System Integration Testing (SIT) and overall quality. Initially, we championed product metrics, positioning ourselves as quality custodians. However, stakeholders, the true decision-makers, favored project metrics, driven by the paramount goal of getting the project ‘across the finish line’.

In another project centered on Software as a Service (SaaS) product delivery, we aimed for balance by incorporating metrics spanning product, project, and process dimensions. We diligently communicated quality risks and agreed upon metrics. However, our client, entrenched in the business domain, grappled with testing metrics’ jargon. Furthermore, our collaborating vendor didn’t fully embrace the defined metrics and their relevance.

Through a fusion of theoretical knowledge, insights from our own implementations, and wisdom borrowed from industry peers, we uncovered ‘the good’. I invite you to join me in this enlightening discussion, where I will be sharing the insights and discoveries that have propelled us toward meaningful and effective testing metrics.