Bloggo back to the blog
Estimating SOA and Web Service Testing Phases-->
A few people have been asking us recently what factors influence SOA and Web Service test phase estimation. We’ve gathered some thoughts here for you on this topic.
Firstly, you’re going to need a tool to execute your tests. The power and capabilities of the tool will greatly influence how long the testing will take. If the tool is cheap (or free), like some open-source offerings, testing estimation can be a fraught process. There can be so many unknowns with free test tools that the margin of error around your estimations will have to be very wide. The more established and powerful the tool is, the more confidence you will have around your estimations.
Put simply, testing can take far longer (and therefore cost far more) using open source tools.
We recently built a test framework for common customer scenarios in a SOA project using Green Hat’s GH Tester. The framework was built before lunchtime on the first day. After that it became more of a traditional data preparation exercise, some of which the tool can help you with and some of which it cannot, depending on how thorough the service design has been. But one thing was very clear after that initial half day of scenario creation, we then had a solid understanding of how long testing preparation would take.
Here are some factors which can influence the time it takes to create test cases and thus influence your estimations:
• How strict are the schema definitions for the services under test? Schemas that have all of their fields as “text” will be much harder to test than schemas that have rich restrictions (such as integer between 0 and 200, regex and so on) as the schema can be used to help derive test cases
• Does the service hosting platform (e.g. an ESB) validate requests against the schema definition before passing the request to the service? If it does is there is no need to do any negative testing, as the service will be shielded from all these bad requests?
• At Green Hat we use a term called incremental integration testing where the various different components are introduced gradually, to control risk. This requires the use of stubs for the components/systems that are not available or may be out of your control. Our more advanced customers publish these stub definitions alongside the service definitions, giving you much of what you need without needing to create the stubs yourself. If you need to build stubs from scratch, this will take more time but again, good tooling will reduce much of this. You’ll need to know what combinations you plan to test with which will be decided by the estimated delivery dates of the services that underpin the business process
• Are there any third party systems you need to coordinate or data preparation exercises which need to be done? These estimates will not be any different from other forms of testing, but make a special note of those things that are out of your control
• Estimating the duration of end to end testing will depend largely on the data combinations that are to be passed through the system. Again, tooling can help create these. The actual estimate will depend on the number of different process paths to be tested and the amount of coverage required
• Finally some thought should be given to performance testing. Unlike GUI performance testing, many SOA tools allow the same functional tests to be used for performance testing, cutting down on development times required for performance testing and allowing it to be done earlier, even at the unit stage. This means the estimates for this part of the process will be much less.
This is not an exhaustive list of factors but should get you pointed in the right direction and provoke some more discussion.
Remember: SOA testing benefits from quick iterations and an agile approach. So, you may find that you are revisiting (and refining) your estimations more often than on a typical testing effort. That’s one of the agile mantras – “Always be planning”.