Bloggo back to the blog
Part 5: …. Zero …… Lift Off (!) ….. Almost-->
Part 5 of 7: Implementation day arrived, and the series of five articles (of which this is the fifth) is being extended to seven episodes, to include the aftermath. The software was implemented and has not proved to be an unqualified success. The project team genuinely were ready. But like a butterfly looking at the empty chrysalis it has just emerged from, there can be no going back. It has to be ‘fix forward’.
The preferred implementation days are Tuesdays and Thursdays, and as well as being fixed in the collective memory of the project team for 3 months as a target, May 14th has been in the overall IT schedule for several weeks. It is not that it could not be changed, but was there as an indicative date for the code promotions. The key was achieving user sign-off in time to finalise the date – described in the forth part of this series.
The software was ‘good enough’ to give business benefit, and as this is not a daily business activity solution (but instead, giving the business users high-level information to aid and assist in long-term decision making), dispensation to undertake the code implementation during the UK business working day was requested and granted. However, ours was not the only implementation of the day, and it is a bit like being in a dentist’s waiting room prepared for ‘the call’. Code promotions, with several different people involved, including a DBA located in Switzerland, were finally achieved 2 hours later than hoped for, at just after 14:00.
Some small hiccoughs in the implementation meant that one part of the first schedule failed, but was resolved within 10 minutes, to enable the first data load to complete by 15:30, so it was on with the next, and then the difficulties began. We found two key differences between the way that the overall data solution works in PROD when compared to test environments. The second data load hit a problem, with the Informatica data load crashing, eventually diagnosed as being a blank input line on the spreadsheet. Prior to data loading, a generic Perl script converts any spreadsheets to comma separated variable (csv) format, and in PROD, a null line at the end of a spreadsheet for one of the data feeds was appearing as “,,,,,,,,”, whereas in all test environments, this was not present at all.
The short-term solution was to edit input spreadsheet by converting to CSV, checking to see whether there was this ‘blank’ line at the end, and if so, removing this in the spreadsheet, converting to CSV to see that it was removed, and then saving as an XLS file, Fiddly but perfectly possible as a short-term solution, and by the end of Wednesday, 16 months of the 24-months of 2011 and 2012 had been loaded for the one monthly feed requiring historic data to be inserted. The first month had been checked and it was correctly present on the database, so loading was set to continue as fast as possible.
Most of the delivered solution is booked into the configuration management tool, but two parts are not, for historical reasons, and “it is not possible for this technology”. In both cases, the components are developed by geographically remote people who are not part of the core project team. One of these is the use of Business Objects to produce ‘pushed’ reports, and a check list was provided to the solution provider, to check certain aspects before code promotions (either from System Test to UAT, or UAT to PROD). From where I am sitting, it was as if the solution had to be developed afresh rather than just pointing to a different Business Objects Universe. It came as no real surprise when the reports were not produced at the first time of asking. This critically masked the second key difference between the test environments and PROD. Data was being loaded into the database twice! (For every data-load except the first one – which had been thoroughly checked using SQL back-end queries).
The double loading of data was identified at 16:30 on the Thursday, and has been referred to the database tool provider to advise and provide a solution. This is a big difficulty, and something of a set of buffers. However, some data has loaded correctly, and the reporting layer enables information previously loaded to be viewed for two of the vertical market segments.
The end of the week had seen software implemented, and it is easy when concentrating on things that are not correct (as testers do) to miss the positives. The UAT test completion report was produced and signed-off, and the User Guide issued (both my documents). Further amendments to presentation aspects of the reporting layer have been successfully accomplished, and user sign-off almost achieved on the reporting aspects (“the pen is hovering over the paper”). Some defects were closed and small parts of the Business Verification plan ticked-off. It has not been an unqualified success (!!), but further stakes in the ground towards success. And in six months time, when we have ‘fixed forward’ – as undoubtedly we will do as a project team, the small difficulties this week and for the next few weeks will seem like a distant memory. The figures show that more defects were closed than were raised – at least one bit of good news
One thing that we do as a project team is track defects fastidiously, and do not pre-close defects. If something was raised against UAT, it can be closed when tested in this environment or later (i.e. UAT or PROD, but not against System Test). Many of the items raised this week were against PROD, so even if there is a software solution available, the defect will not be closed until the solution has been deployed and shown to be correct in that environment. That can sometimes inflate the ‘open’ figure – but is a true reflection.
It was the unexpected that tripped us up – “the expected you can cope with, but on-one expects the unexpected” – Brett Gonzales. If we had known about the differences between the test environments and PROD, it would have been factored in, solved and delivered. The root cause of both differences is still a mystery. As it was, the project team have to cope with the unexpected. It is what has kept us busy, and running to stand still, not quite sure what the next hour would bring. The declared intention has always been to fix forward, and as it is, some real benefit has been delivered. Not as much as was hoped at this stage, but more will accrue in the next weeks and months. Like the clairvoyants tea party, we catered for the expected – that was within our control. You cannot control the uncontrollable! It has been a true “Agile” implementation! (pun intended)
Not an unqualified success, which is why this is a story to be continued. I would not like to leave you in limbo, as if were, so expect two further installments, one upto the end of May and one around 4 weeks later. It will only be in the last report that an element of perspective will emerge. At present, I am too immersed in the detail of the parts that need temporary or permanent solutions to have that sense of perspective.
Peter Morgan is a testing professional who has been involved in the ICT industry for more than 30 years, and worked in the freelance marketplace for much of that time. His time has sometimes moved from testing to ‘development’, but he would add “always using the mindset of a tester”. He is passionate about testing and a firm advocate of testing qualifications. An enthusiastic speaker and author, Peter tries to base his output on hands-on experience, attempting to relate fine sounding ideas back to how it will affect Joe or Jane Tester in their everyday working lives in the war of attrition that we call software testing.