How do you measure the test coverage for each software testing level? (2024)

One should be aware that Code Coverage only provides information about whether statements are executed during test. It provides no information about the quality of the tests themselves. Therefore, also for unit testing test techniques like boundary-testing or equivalence partitioning should be considered.

Code coverage tools can also be integrated and measured in the CI pipelines and thus enforce the testing coverage percentage matches the expectation before merging commits or pull requests.

The reason why we do integration testing is that the software or the product might be working as expected when used as it is. But when it is interacting with external software or a component, there are high chances of not going in your way.To prevent this kinda issues in the real-world, we do Integration Testing.

Measuring requirements coverage for me is tricky and might be misleading. When is a requirement considered tested and thus covered? In case we have 1 test case testing the requirement? Or maybe that's not enough so we need 3 test cases?Or maybe it is dependent of the complexity of the requirement? Do we need happy-flows, alternative-flows and sad-flows tested to be covered?In my opinion, requirements coverage provides information which can be used to form an expert-opinion whether we tested sufficiently. It is not a hard number showing whether you tested sufficiently.

Measuring tests cases against requirements can be misleading as requirements can vary based on their complexity. One test can cover a single requirement and multiple tests can also cover single requirement (complex). Test reviews from SMEs can also help to some extent if requirements are fully covered by tests. However it is still a manual effort and cannot be done for growing requirements. Still Struggling to find automated approaches to measure functional test coverage w.r.t requirements and would like to see other contributions!

Another approach to consider is to create a Requirements Traceability Matrix that maps each requirement to the corresponding test case in the acceptance test suite. This matrix serves as a visual representation of which requirements have associated tests.

How extensively you would like to or need to test is dependent of the application of your software. In case you are building safety-critical systems in which malfunction may lead to injury or even death, you need to test extensively and focus on corner cases (alternative flows, sad flows) as well.When defining a test strategy you might consider to have a look at the Agile Testing Quadrants as defined by Lisa Crispin and Janet Gregory.

Another level of testing is Chaos testing or Chaos Engineering. This testing focused on testing a system's integrity by proactively simulating failures and disturbances in a given environment. The goal is proactively identify weaknesses and vulnerabilities in a system's architecture, infrastructure, and components before they can cause significant issues in a production environment. To measure the test coverage for Chaos testing, create a coverage matrix that maps different chaos scenarios to specific components or subsystems of your system. After that, ensure that the chaos testing tool or framework provides sufficient coverage across different failure types and occurs in a way that reasonably represents real-world scenarios.