Does your development team spend too much time waiting on their test suite to run? Do they constantly rerun the test suite after failing tests because “rerunning magically fixes it”? If your developers have these problems, there’s a good chance their test suite doesn’t follow the test automation pyramid.
The test automation pyramid is an important concept that all software developers should be familiar with. It’s a framework that can help guide the development team into producing a higher-quality product. The test pyramid also reduces the time it takes developers to find out if they introduced a breaking change. As well, it can lead to a more reliable test suite.
This post is going to dive into the test automation pyramid and introduce some strategies for managing it.
Before we dive into the test automation pyramid, let’s define test automation. I’m not going to reinvent the wheel, so I’m going to borrow the definition from this blog post: “Test automation is the practice of running tests automatically, managing test data, and utilizing results to improve software quality. It’s primarily a quality assurance measure, but its activities involve the commitment of the entire software production team.”
Test automation can reduce the amount of time developers spend testing their software because the environment can be prepared and cleaned up automatically. Not only that, but it can report the results of the test run to be used for future insights. This can help developers debug problems faster.
So now that we know what test automation is, let’s see where the pyramid fits in.
What’s the Test Automation Pyramid?
The test automation pyramid represents the various types of tests and the frequency at which they should appear in the codebase’s test suite. It’s all about giving the developer immediate feedback that code changes don’t break existing features. The test pyramid has three distinct sections. Unit tests are at the lowest level, integration tests make up the middle tier, and end-to-end tests form the top. The test pyramid helps developers deliver quality software.
Why do we need to make a distinction between the types of tests in the codebase? Let’s answer this question by breaking down the tiers. I’ll also define some common tooling in each of the levels and introduce some tips for managing the test automation pyramid.
Our test pyramid needs a strong base, and that’s where unit tests come in. This means they’re the most common type of test in the test pyramid.
What’s a unit test? A unit test focuses on testing a very small component or piece of functionality of the codebase. Its goal is to validate that the unit behaves as expected in isolated conditions. This could be an individual function or a class. Developers should focus on testing a number of scenarios ranging from happy path to error handling.
The unit test suite should run quickly because they’re the largest subset of tests in our pyramid and will continue to grow in number as new features are added. The unit test suite should be run any time a developer makes a code change. This gives the developer immediate feedback on whether or not their new code broke anything in the codebase. A fast unit test suite encourages developers to run the suite more often, which cuts down on the time spent debugging problems.
One way to build a strong unit test suite is through the practice of test-driven development (TDD). TDD can naturally lead to a strong unit test suite because it requires a test to be written before any code. As new features are added, new unit tests are created.
Unit tests are great for testing small pieces of a codebase. However, unit tests aren’t representative of a deployed application. And they don’t test the interactions of the app with the outside world. This is where integration tests come in.
Unit tests aren’t enough to ensure the quality of a codebase, so now we’ll introduce integration tests. Integration tests make up the middle tier of our test automation pyramid. This means that integration tests should not appear as often as unit tests. Integration testing can be an overloaded term in the software development world, which is prone to confusion. I’ll define integration tests as tests that validate interactions with external components. Let’s unpack that a bit. External components can be anything that lives outside of your application, but that your app still depends on.
For instance, databases are a very common external dependency in software applications. It’s important to test that your application interacts with your database as intended. Database integration tests should ideally run against a real instance of a database similar to the production environment. However, this requires managing the database locally or in the build pipeline. Docker can make this easy to manage because it offers images of many of the popular databases including MySQL and even Microsoft SQL Server.
Furthermore, the integration with external services should also be tested. These types of tests can also be called service tests. Service integration tests should focus on the interaction of your code with the external service such as a RESTful API.
So why do developers even bother with unit tests if integration tests show our code is functioning with the external dependencies? Because these dependencies are outside our app, the feature we’re testing requires communication to the dependency. This will make a call to the database or web service, which can cause the integration test to run slower compared to unit tests. Furthermore, if we’re testing an integration with an external web service, it’ll require a preproduction environment capable of testing against. Finally, it may be difficult to test each scenario from an integration point of view, such as error cases, because we cannot control the responses from the dependency.
Finally, we’re at the top of the test automation pyramid. At the end of the day, we want to make sure our entire application is functioning as expected, and that’s exactly what end-to-end tests help with. End-to-end tests are what they sound like; they test that your application is working from start to finish. In other words, they test the integration of the front end with the back end.
I like to think of end-to-end tests from the perspective of the end user. How would a person interact with my app? And how can I write a test from the user’s perspective? Let’s dig into an example. Our application more than likely has a login page. And we may want to test that the login page is working. This would require the following steps:
- Find the username text field and type the username.
- Locate the password text field and type the password.
- Click the submit button.
This could be a very time-consuming and tedious process for developers to test their code change each and every time. Not only that, but different developers may have different strategies for this manual test. So how can we make this process easier to manage and more reproducible?
Selenium is a testing framework that automates interactions within a browser. Selenium provides a domain-specific language that allows developers to write tests that interact with a web app running inside the browser. We could write a test with Selenium to follow the steps of the login process. This also shows that the integration with the UI to the back-end API is working together.
End-to-end tests are at the top of the test automation pyramid because they can be slow and very fragile. And like integration tests, they may depend on external dependencies that may be unreliable.
Let’s recap what we’ve learned now that we understand the concept of the test automation pyramid.
- The test automation pyramid is a framework that defines various types of tests and the number of times they should appear.
- Unit tests form the base of the test pyramid. They should be frequent, and they should run fast.
- Integration tests are the middle tier of the pyramid. These tests focus on interactions of your code with the outside world, such as databases and external services.
- End-to-end tests top the test pyramid. They’re written from the perspective of a user and should test that your entire application is functioning from front end to back end.
Now that you know what the test pyramid is, do you think your development team’s codebase has a strong pyramid? When a team has a robust test pyramid, coupled with test automation, it can help the development team ship features faster and with higher quality. Learn more about test automation and how to choose a test automation platform that fits your development teams’ needs.
This post was written by JT Wheeler. JT got his start working as an engineer in the manufacturing industry. From there, he moved on to become a developer at a software consulting firm. He’s written software that spans a number of industries ranging from industrial automation to inventory control to financial systems, and he’s passionate about learning and teaching new technologies and software practices.