The Dragons of the Unknown
There is a phrase used by people working with complex safety critical systems; “jousting with dragons” or “facing the dragons at the borderlands”. It comes from the practice in olden days of map-makers who would fill in the gaps in their knowledge of the world with monsters and warn “here be dragons”.
The dragons meant dangerous, unexplored territory. Safety critical systems are complex and dynamic. Safety experts know that they don’t know how these systems will behave. They can’t even predict the system boundaries. They just know users will stray into the unknown, to face dragons and danger.
The safety critical community has to deal with the world as it is. If they don’t do their job properly then people get killed. They have to confront the reality of these highly complex socio-technical systems and be brave enough to say “I don’t know” when the frightening truth is that they can’t know. These systems can’t be understood or specified completely in advance. They will always fail in some way. In the hands of humans they will fail in ways it is hard to imagine in advance. It is vital to gain a better understanding of how people will use them, what happens when they inevitably fail, how people can recover, whether recovery is possible. I will refer specifically to safety critical work around the Safety II framework.
Increasingly problems faced by the safety community will confront all testers. Every ecommerce site is complex and dynamic, however simple it might appear at first sight.
If we can’t handle complexity as honestly as the safety experts and face the dragons of the unknown then we could turn testing into a dreary, pointless, bureaucratic job, doing little of value, and performed by whoever is prepared to do it for least money. We have to be brave enough to know when “I don’t know” is the right answer. I will draw heavily on my experience working with complex systems as a tester, developer and auditor.