There are many ways in which a testing strategy can go wrong. One of them is simply not testing all of the parts that need testing. You can fight this problem by measuring and improving your project’s test coverage.
Simply put, test coverage is a measure of how well your existing tests protect your app against regressions. In case that definition isn’t clear enough, don’t worry: we’ll start the post by expanding on it, while also explaining how test coverage differs from code coverage.
After that, we cover the main benefits of test coverage. Finally, we give tips on how to make sure your application is well covered, before parting ways with some final considerations.
Test Coverage 101
The central part of this post, as its title makes clear, is the “how” of test coverage. But first, we need to understand the “what” and “why” as well.
Test Coverage Fundamentals: Definition and Examples
In the introduction, we defined test coverage as a metric. It measures how much testing is being performed by your test suite. But there’s another way to look at test coverage. Consider it a technique to ensure that all parts of the application are being exercised by running the tests.
Suppose your application has 50 features, and that when you run your tests, they exercise only 37 of those features. In this case, we could say that your tests cover 74% of the application.
Another example: Suppose your application is a personal finance manager. Among its features is the ability to export your transactions to four formats: .pdf, .xls, .csv, and .ofx. It can also import transactions, but in this case, the formats supported are only .xls, .csv, and .ofx. To achieve good coverage, you’d have to test every one of the importing/exporting scenarios above.
An essential characteristic of test coverage is that you can consider it a black-box technique. That is to say, when performing test coverage, we don’t care about code or how the system is structured internally. Instead, test coverage is more of a high-level approach, concerning itself with features and requirements instead of implementation details.
The Benefits of Test Coverage
Analyzing test coverage on a frequent basis can inform your testing strategy. Understanding the features that are covered and uncovered helps you to assess risks. Specifically, you can:
- Identify gaps in testing. The main benefit of test coverage is that it allows you to identify areas of your application that aren’t covered by current test cases.
- Assess the quality of existing tests. Having a high number of test cases but large areas of your application untested may indicate that the test cases aren’t well structured to cover the features of the application.
- Find useless test cases. By analyzing test coverage, you could reveal test cases that verify already-tested features and thus represent duplicated effort. Such test cases can then be eliminated, making the code lighter and bringing down test run time.
What About Code Coverage?
Time to clarify a common misconception. People often use “test coverage” and “code coverage” as if they were interchangeable. They aren’t.
Test Coverage vs. Code Coverage
Test coverage, as you’ve just seen, is a measure of how well your tests exercise the different areas in your application. We could say it’s a qualitative measure of your existing set of tests. And remember that we can consider it a black-box technique: it doesn’t concern itself too much with code and internal implementation details.
Code coverage, on the other hand, is a metric most often associated with unit testing, and it measures the portion of a codebase exercised by test cases. With code coverage, you are testing the functionality of the code as well as the implementation details.
Types of Code Coverage
The most straightforward type of code coverage is line coverage. It checks whether tests cover a given line. It’s represented by the percentage of lines covered by tests.
Line coverage information isn’t sufficient, though. You could have a situation like in the following example:
if (x > 5) {
// logic if if-statement evaluates as TRUE
console.log('x is greater than five! YAY!');
}
console.log('This line will always be executed.');
Suppose that you have a test case that exercises the scenario where x is indeed greater than five. In this case, the tests would exercise all of the lines above, and you’d get 100% of coverage. However, you don’t test the scenario where x is less than or equals to five. That’s where branch coverage would be useful.
There are more types of code coverage available, including function coverage, block coverage, and statement coverage.
The Limitations of Code Coverage
In the previous section, we talked about some of the limitations of code coverage, particularly of the line coverage variety. The problems go beyond that, though.
The main problem of using code coverage as a metric is that it doesn’t tell you anything about the quality of the tests. While low code coverage is likely a problem, high code coverage isn’t necessarily great. The most extreme example of this would be having 100% coverage but no tests performing any assertions. In this case, your tests wouldn’t be testing anything.
Another limitation of code coverage is that it only works with code-based tests, particularly unit tests. While this type of testing is undoubtedly critical and can provide valuable feedback, it’s certainly not enough. For instance, customers won’t care that your underlying API works perfectly if they have to see and interact with a faulty UI. That’s why UI testing is also essential, but code coverage won’t help you there.
Make no mistake: code coverage is a valuable metric if you use it wisely and remain alert to its shortcomings. However, when it comes to having a big-picture view of how well your application is being tested, test coverage is what’s got your back.
How Do You Perform Test Coverage?
One of the hard things about test coverage is that calculating it might not be as straightforward as calculating code coverage. Techniques for coming up with your test coverage vary. Even the definition of what it means for a feature to be “covered” varies, and depends on the team and the type of software being created, among other factors.
We have another post featuring different test coverage techniques. We recommend reading it, but the gist of it is what follows.
Product Coverage
Product coverage consists of approaching test coverage from a product perspective. That means not only verifying the functional aspects—i.e., features behave as expected—but also the non-functional ones, such as performance, usability, and accessibility.
Risk Coverage
Risk coverage helps you assess the risks an application opens itself to. Employing this technique consists of writing down the risks inherent to the application and ensuring there are test cases covering them.
Requirements Coverage
Requirements coverage is certainly the most well-known and arguably the most important test coverage technique. It consists of listing out the requirements for the application and making sure you have tests covering them.
How Do You Ensure Test Coverage Is Good?
- Create a comprehensive testing strategy. It should take into account the application’s requirements as well as the testing methods you’ll be employing.
- Create a checklist for all of the testing activities. Based on your testing strategy, the next step is to create a list of actual tasks that need to be carried out. Take into account the different types of testing, both manual and automated, the size and experience level of your team, and the type of application you develop.
- Prioritize critical areas of the application. When resources are scarce (and when aren’t they?), you need to favor a risk-based approach to testing. Favor areas of the application that are both critical and have a high probability of having problems.
- Create a list of all requirements for the application. This will be invaluable when performing both product and requirements coverage.
- Write down the risks inherent to the application. This is required to perform risk coverage.
- Leverage test automation. That way, you’ll reduce overall testing time, free professionals from doing repetitive, tedious, and error-prone activities, and be able to cover many more portions of the application.
Improve Test Coverage Through Test Automation
The last item on the list above is “leverage test automation,” and that’s by design. We believe that test automation is no longer a “nice-to-have” but a “must-have” in a modern software development process. There’s no other way to achieve high confidence in the software you release while still being able to ship it on time.
A modern, AI-powered test automation tool like Testim can help teams create reliable test suites that increase test coverage without making test maintenance a burden.
