go back to the blog

Part 6: I-day + 2 weeks – the smoke begins to clear

  • 06/06/2013
  • no comments
  • Posted by EuroSTAR

The defects statistics I have been showing have not been telling the whole story. These have been summary figures only, and just like the Data Warehouse application being implemented, it is possible to ‘slice and dice’ figures in different ways. The mid-May implementation is both building upon previous software implementations, and part of a larger process – so as the ‘current’ work-phase, there are previous and future parts. There are also different environments, not just the PROD environment. So to attempt to put defects into context, the total for open defects is given, and how these split between both environments (‘Test’ and PROD) and the overarching macro-project (Previous, Current and Future). Sometimes the project ‘stage’ is a little arbitrary – previously loaded data that caused no problems with the existing solution but prevented the loading of data in the current stage – is that ‘previous’ or ‘current’?

Items that should have been included in mid-May (part of the logical set of deliverables but not in the actual items delivered) are classed as ‘current’. It was always planned that there would be future implementations; to deliver items that could not be squeezed in for mid-May; to resolve any newly found issues and to begin work for the future stages. Two such deployments have been planned for some time, one at the end of May – already implemented – and one in mid June. Testing is in progress for the latter of these, with the items for inclusion not quite finalised.

So, what of the implementation? The major problems have been data. The double-loading of transactions turned out to be some duplicate data already on the PROD database – each transaction finds what should be unique reference data, by both item key and effective dates. When the reference data is not unique, two transactions get loaded for the same single input (a simple explanation). Duplicate transactions have been removed, but the offending reference data is owned by another business unit within the shared data warehouse – so may take some days to tidy up. Until that has happened, not further transactions will be loaded. However, all data up-to and including the end of March 2013 has been loaded. The business can begin to look at the data, and see emerging patterns of activity – either not possible hitherto, or not possible to compare similar output for the five different market segments. Information in the warehouse for this stage is used for long-term analysis, so the immediacy of data is not important in the short term.

Data problems have also been found with upstream business processes (as was expected) and some further amendments were required with the presentation aspects of the reporting layer. Any problems are shown on a large chart, together with the data that has been loaded, and the progress towards completing the business verification (BV). There are nine data load schedules, each run on demand, with the use of a specific trigger file in a polled directory depending upon the individual schedule. Some of the trigger files contain specific data to load (“data content specific triggers”**), whilst others are purely an indication to run the appropriate schedule (“data content agnostic triggers”**). [** I am grateful to Brett Gonzales for this terminology]. Users had been starting schedules successfully, and all but one of the nine schedules started and completed as requested. Progress has been made both in terms of BV and the loading of data files and this is all shown on BVP’s (Big Visible Pictures).

Phantom blank data rows at the end of transaction data files, an issue in deployment week and worked around, are now being ignored by the Informatica ETL processes (a code change implemented at the end of May), but the reason why seeming the same Perl script behaves differently in test environments and PROD is still under investigation.

Implemented? Yes. Successful? A qualified ‘Yes’. Completed? No. Fixing forward and moving on to the next stage? Yes – the first small part of the next major delivery will be implemented in mid June!


Peter Morgan is a testing professional who has been involved in the ICT industry for more than 30 years, and worked in the freelance marketplace for much of that time. His time has sometimes moved from testing to ‘development’, but he would add “always using the mindset of a tester”. He is passionate about testing and a firm advocate of testing qualifications. An enthusiastic speaker and author, Peter tries to base his output on hands-on experience, attempting to relate fine sounding ideas back to how it will affect Joe or Jane Tester in their everyday working lives in the war of attrition that we call software testing.

Peter is also one of the 2013 EuroSTAR Community Leaders.

Blog post by

go back to the blog


Leave your blog link in the comments below.

EuroSTAR In Pictures

View image gallery