Bloggo back to the blog
Requirements fulfilled but is it fit for purpose?-->
Now the clever reader will say; “well that’s not a problem because they should just do agile development then the problem would go away”.
But agile is not the silver bullet – not the answer to all evil. If the nature of your project or maybe the nature of your customer makes it impossible to get the involvement through the entire development lifecycle, if they can’t supply a product owner and cant be involved in task breakdown, feature workshops and so on – if the customer can’t supply users to be there for testing during the development, then agile isn’t the whole answer.
So what to do?
Well you could try to do something like we did; find your own users and let them be involved in feature workshops and even more – let them test the best way possible; by letting them do their daily work.
The system in question for us was a large scale army command and control system being used in military headquarters for planning and mission execution – a domain hard to grasp for both developers and testers in the project, a domain far away from all we know.
The first thing to do for us was to get our own product owners since we as a product didn’t have just one customer but many – and customers that were not right around the corner geographically since it was nations all around the world. We had domain experts for all military products, former army officers that has actually been out there – in the real world. But that wasn’t quite enough for us, we took it one step further; we found our very own operational users! With help from the domain experts we identified a number of active officers and non-commissioned officers and enrolled them as operational testers.
The use of these new testers was two-fold;
1. Having real users actively involved in feature workshops, prototyping etc. to get the users perspective on new functionality BEFORE we coded it.
2. Having real users or operational testers as we called them – executing test, but in “opera-tional way”.
In order to test “the operational way” we needed to change the way we tested away from the traditional scripted test specification or the freestyle exploratory testing – to a test type where we enforced a focus on the operational usage of our system.
So our domain experts in cooperation with active army officers designed a real army exercise with all the documentation and information a normal army unit would have when participating in an exercise (operational orders, plans, overlays), and we defined a test environment that fitted that military organisation. Then we enrolled the operational users once again, gave them a standard training course in the system as they would get normally – and then we conducted an army exercise.
With that I mean;
• No test specifications
• No requirements
• No telling them what to do and how
• Just handing them the operational background and the system and letting them do what they do best – army stuff.
We participated with different roles;
• Test manager; ensuring that findings were recorded and logfiles kept (and bring coffee )
• Domain advisor; support when operational vs system problems occurred + getting new operational knowledge
• Testers; acting as assistants to the operational users, being the foot soldiers
The test was done over an entire weekend and turned out to be a great success. That for several reasons; we got the system validated seen from the operational user’s perspective – they had findings we wouldn’t have found because we don’t have the detailed knowledge about operations in all aspects, but also on the positive side – they had “thumbs up” for the functionality we had chosen to implement. We got new knowledge about army operations, and “we” were both the domain experts and the testers. And last but not least we now have a group of ambassadors, soldiers that knows our system and aren’t afraid of using it – who can tell the story and help making implementation “out there” a success.
So all in all a great experience – and now you may say “what can I use this for – I don’t have an army system?”
This concept can be implemented on ANY system that has a workflow; if you have a medical system then let the medical personal do their morning round. If you have a freight system – follow the package from start to end, do the WHOLE workflow, not just as individual use cases (such as create package, register package). But don’t do it yourself, get the real users to do their thing on your system, people who don’t know about why the system is implemented as it is – don’t know about the corners you may have cut.
With activities like this we get into the core of what it is all about; what does the user really need in order to be able to do their job? How can our system in the best way possible support their tasks – and thereby we find the holes in the system before it “hits the user”.
But remember; this is not the silver bullet, the first part I mentioned (feature workshops etc) is essential to catch the really nasty holes in requirements – the missing fit for purpose. The operational test is again just another tool in the test manager’s toolbox……………but a good one