Continuous Testing (CT) is the process of executing automated tests as part of the software delivery lifecycle in order to obtain timely feedback from businesses in order to save on costs and time. However, an important question that bothers many testers is “When should one stop testing?”
It’s a question that is normally answered in advance as part of the exit criteria for a test plan. Let’s consider the real-life possibilities:
- When we run out of time
- When the testers and/or the test environment are all re-deployed for another test
- When the project budget runs out
- When we have reached an acceptable level of risk
- When all the defects have been found.
The test professional’s view would be d) – when an acceptable level of risk is reached. Too often, project and program demands mean that we are forced to compromise and end up being corralled into “accepting” running out of time, or the test environment being redeployed, or the project budget running out. However, the last option of having all the bugs found sounds impossible – right?
So, in the real world, what happens after we stop testing upon running out of time or having reached an acceptable level of risk? In these cases, testing doesn’t stop. Whether we have done this consciously, or not, “testing” continues but is carried out by the end-user community and any defects or bugs found at this stage are fed back into the IT help desk. But is that optimal? Are we asking the right question? Instead of “When should you stop testing?” shouldn’t we ask “Why should you stop testing?”
The real goal ought to be achieving an acceptable level of risk and testers can do so only if they adopt a policy of continuous testing. We conducted a global survey to outline the challenges and potential approaches to transforming test practices in the age of agile and DevOps. We surveyed 500 senior decision-makers in corporate IT functions, working for companies and public-sector organizations across eight different countries.
Out of the 500 surveyed respondents, 32 percent acknowledged having fully adopted continuous testing. Twenty-five percent of those surveyed stated that a vast majority of their teams practice continuous testing.
Enterprises are now embracing continuous testing by adopting test automation for various quality assurance (QA) tasks along with the move towards agile and DevOps.
While organizations today have adopted agile and DevOps methodologies, testing remains the missing key needed to unlock the complete transformation of the software delivery lifecycle. Our Continuous Testing Report (CTR) 2019 looks at some of the key trends, challenges, or gaps in the areas of testing and the remedial measures required that organizations can undertake to improve.
