The actual execution of a test can start at the point when the test object is deployed to the relevant environment. In a CI/CD pipeline the automated test will startautomatically as part of the process. In other situations, a team member must start the execution.
Definition
Test execution is the execution of tests by running the system under test and thus obtaining the actual results that can be compared with the expected results to determine whether the tests have passed or failed. This is part of dynamic testing.
The first step in test execution is to make sure the test object is properly installed and all preconditions (such as availability of the right test data) have been fulfilled. This is often done by performing a pre-test (also known as intake test or smoke test).
Definition
A pre-test is testing the delivered product in such a way that it is determinedwhether or notthe product is of sufficient quality to execute a complete test of this product.
When the team has created test scenarios and/or test scripts the test execution will execute many test cases together and investigate the results after execution. When the team does exploratory testing, test cases are designed and executediteratively,and the result is investigated after every test case execution (See figure below).
Explicit and implicit tests
When executing tests, the tester observes the behavior of the IT system. With the explicitly designed test cases there are specified expected outcomes. However, the tester must also observe the behavior of the test object more generally. Specific aspects of the system are often tested implicitly, that is without creating specific test cases. Think for example of a tester that is testing an arithmetic function and while doing so also pays attention to the usability of that function without having explicit usability test cases.
Definition
An implicit test is observing behavior of a test object during execution of test cases that were not designed for testing that specific behavior
Definition
Explicit testing is defining and executing specific test situations and test cases for testing a specific situation.
Teams that automate tests in their CI/CD-pipeline should be very aware of the fact that an automated test only performs explicit testing. So, if there are aspects that would be tested implicitly in manual testing, explicit tests will have to be created to be able to have a valid assessment of the quality of the test object.
Different test varieties have a different focus
A DevOps team will need a lot of variety in testing. Even when all testing is organized and performed together, still the team needs to be aware of this variety.
Executing an API test for example, is very much different from a UX (user experience) test or an end-to-end customer-journey test or a non-functional test. Therefore, it is often wise to organize test varieties separately and to carefully consider which quality risks and business value are best tested in what test variety.
Also, different test varieties will need different test execution schedules. Typically, tests are only executed if related parts of the system have been changed. If, for example, the user interface was changed but the API was not, then only UX testingneedto be executed.
What if the test cannot be executed?
There are many reasons why a test cannot be executed. The test environment may not work, another test may have failed and blocked all following tests, the test data may not be available, the test script may need to be adjusted because of interface changes, et cetera.
The worst reaction to a test not being run is assuming that it would have passed. This may seem obvious, but teams that are working towards a deadline, for example the end of a sprint, may considercuttingsome corners and still deliver. If a test cannot be executed, the exit criteria are not fulfilled and thus the team cannot deliver their changes. Therefore, the team will need to focus on proper maintenance of their test scenarios, test data and test scripts. Continuous testingnot only refers to continuous test execution, but also to continuous maintenance of thetestware.
