As part of improving our understanding of a legacy system, in “The Product Owner’s Guide To Escaping Legacy” I explain how we can go about discovering new information about a system, and how we can keep track of how we are currently testing that functionality. Then, we use that knowledge to be more targeted in our approach to testing.
The goal of testing is to be able to release with confidence. Confidence can come from many different activities and measurements. What we are doing here initially is reducing a particular type of, very visible, activity around testing: the scripted manual regression test. Since in most situations this is where the bulk of the test effort is centered, this will feel like a significant reduction in testing, and an assumed reduction in test coverage. In other words: it’s scary and reduces confidence.
The goal of testing is to release with confidence
The assumption that doing less work in manual regression testing reduces test coverage could very well be true. This is why we repeatedly re-prioritize where we focus our testing and are changing our approach iteratively and not in one swift and drastic change. In reality, though, test coverage is often unknown. Functional test coverage, telling us how much of the functionality of our application is being validated by our testing, is quite difficult to measure. As we’re in the process of extending our knowledge of what the functionality of our legacy application actually is, we start to see what part of that functionality we are actually testing. Initially, that additional transparency will make it seem like testing is not as extensive as we thought it was and that can, again, reduce confidence!
Test coverage is not the goal
That means part of that reduced confidence is due to a much-needed reality check. Increasing transparency is a double-edged knife. We’ve seen that quality is not what it should be, and our level of confidence is being brought in line with what is appropriate for our way of testing. Another part is that so far we have focused on tests that are fairly ‘high level’ and there are certain types of logic and complexity that are not served by that sort of testing. In the book, I explain how to go about testing that sort of functionality, and how to decide what type of testing to use.
The first step in building confidence in testing is making it very visible what we are testing and why, and keeping track of the results. Once we start seeing that the testing is actually working to catch issues, and adjusting it where it doesn’t, confidence will grow.