Regression testing is a specific form of testing that verifies whether a given piece of software suffered regressions after undergoing changes. “Regression” here means “going back to a previous undesired state.” So automated regression testing is nothing more than the process of automatically verifying that the application has not regressed to a previous undesired state.
Ideally, regression testing—whether automated or not—is something you do every time a software application is changed in some way, whether by receiving a new feature, an improvement, or a bug fix.
In today’s post, we’ll define and explain what automated regression testing means. We’ll start by explaining what a regression is, how costly they are for software teams, and why you should employ regression testing to avoid them. Since manual regression testing would be both time-consuming and error-prone, we’ll then proceed to cover the need for automation, offering advice and tips on how to actually implement the technique for your teams.
Let’s get started!
What Is Automated Regression Testing?
Before we dive into the “how” of automated regression testing, we need to clarify its “what” and “why.” What is it, after all? What does automated regression testing mean, and why should we use it?
This section will answer the questions above and more.
Automated regression testing is a form of automated testing meant to detect regressions in an application. “Regression”, in this context, means that the application has regressed back to an undesired state. For instance, an old bug resurfaced, or a previously perfect feature stopped working.
Why Do We Need Regression Testing?
As we’ve just defined in the previous section, “regression” means an unintentional reversion to a previous state. However, something really important was left implicit in the definition: software regression, more often than not, means the application went back to a bad previous state.
Have you ever had the experience of making some existing feature stop working after implementing a new one? What about that nasty bug returning months after you thought you had completely eliminated it? These are examples of regression. The application has regressed to a previous inferior state.
Regression Testing Is Vital in Software
The field of software development is disproportionately prone to regression problems. Each addition or change made by any developer has the potential to cause unexpected problems in areas (supposedly) unrelated to the spot where it was performed. Any non-trivial software project, maintained by a team with more than, say, five people, has an incredibly high number of potential regressions during each release.
Regression testing is nothing more than the execution (partial or total) of a test suite in order to verify that a given application hasn’t returned to a previous undesired state.
Developers should always keep in mind that all the changes they make, no matter how small, simple, or insignificant they seem, have the potential to cause surprising side effects. They can break functionalities that don’t have anything to do with new changes. By performing regression tests, the developer checks that not only does their change behave as it’s supposed to, but also that it plays well with all of the code that was written up until that point.
That’s where regression testing comes in handy. Regression testing is nothing more than the execution (partial or total) of a test suite in order to verify that a given application hasn’t returned to a previous undesired state. If manually done, though, regression testing can be extremely time-consuming and error-prone, which leads us to our next point.
Comparison of Automated vs Manual Regression Testing
I often say that everything that’s automatable should be automated. Meaning, if you can automate a process, then you probably should do so. If you can automate a process but you’re still doing it manually, I’m afraid you’re leaving money on the table. Chances are the manual process is slow, time-consuming, tedious, and error-prone. That implies you’re losing money in at least three different ways.
Manual testing creates opportunity-cost
For starters, you waste money by having well-paid professional performing tasks that could be automated. Then, there’s always the opportunity cost. The people who are performing the tests could be doing more valuable tasks. Such tasks could have the potential to generate way more value.
Manual testing is often error-prone
Finally, since the manual process is error-prone, then people are bound to make mistakes, which will result in losses. Automation solves that since a test script or automated test case is guaranteed to do the same thing 100% of the time.
Automated Regression Testing: How to Put It Into Practice
Now that we’ve defined some concepts, it’s time for some practical tips on how to actually implement regression testing. There’s good news. If you already write some kind of automated tests for your application—e.g. unit tests—then you’re already performing regression testing without even knowing about it.
Regression testing is not a “new” category of automated tests. It’s not an alternative to unit tests or integration tests. On the contrary, your automated tests —unit tests, integration tests, and similar—written from day one in the project can and should act as regression tests. After each new change to the codebase, just re-run all of the relevant tests on the suite to ensure they’re not failing.
Automated Regression Testing Best Practices
Additionally, there are some best practices you should implement when adopting regression testing, regardless of the specifics of your implementation:
1. Adopt a Test Management Software: This is vital for all projects that are larger than a one-developer toy project. You’ll have, realistically, way more tests cases than you’d be able to track and manage individually. Luckily for you, there are lots of test management tools at your disposal.
2. Keep a Testing Schedule: You should maintain a strict testing schedule throughout the entire project. This will ensure the final project is thoroughly tested. Additionally, the schedule will encourage the team to adapt to a frequent testing regimen.
3. Write a New Failing Test for Every New Bug Found: Imagine your code has an unequivocal, reproducible bug. But all of your tests are passing. That means that either the current tests are wrong, or your test suite is lacking tests. If you find yourself in this scenario, write a new failing test to document the bug.
4. Categorize All of Your Tests: You should split your test suite into smaller categories. Your test management tool will most likely provide you with the ability to categorize your tests. That way, your team members can easily identify each kind of test.
Your regression test suite should be a living thing.
5. Keep your test cases up to date: Your regression test suite should be a living thing. You’ll be constantly adding new tests as your application grows. Also, you should delete test cases that are no longer relevant, and correct tests when you find out they’re wrong.
6. Care about readability and maintainability of your tests. When it comes to code-based tests, you have to treat them with the same care you treat your production code. Test code is just code, after all. It might have bugs, it might be hard to read, it might be hard to maintain. So, give your test code some love: keep it readable, refactor it when necessary, try to eliminate unnecessary duplication, and so on.
7. Make sure your tests are in your pipeline. You want your tests to run as often as possible, which means including your regression suite as a step into your pipeline is a great idea.
Challenges in Automated Regression Testing
It’s not all a bed of roses when it comes to automated regression testing. There are significant challenges you must be aware of, and now we’ll list some of them:
- Test maintenance. As your application grows, the maintenance of your test suite, if not done effectively, can become a burden.
- Fragile tests. Depending on your choice of tools, you might end up with a test suite that fails all of the time, or for intermittent reasons, eroding your confidence in the tests.
- Execution time. As your application grows, the time required for all tests to execute could grow dramatically as well. This is particularly true for regression suites that contain a lot of end-to-end tests.
- Environment consistency. It’s often hard to keep environments consistent, ensuring that your test environments are as close to production as possible.
Integration of Automated Regression Testing into CI/CD Pipelines
I’ve touched briefly on the importance of adding your automated regression suite to your pipeline. Let’s now expand this thought a little bit.
Ideally, you’d have all of your tests run every time someone pushes to main (in case you’re using trunk-based development) or merges a PR (in case you’re using a pull-requests based workflow). That might not always be feasible, because some of the tests in your suite might be really slow (think of end-to-end tests that exercise the application via its UI.)
A good compromise is to have your quick tests more often (for instance, every push or merge) and your slower tests less often (say, once a day, or before any planned deploy.) Since you’ve already categorized your tests (see best practice #4) that shouldn’t be that hard to do.
Maintaining and Updating Automated Regression Tests
I’ve also touched briefly on the importance of keep your tests up to date, readable and maintainable. Let’s also expand that a little bit.
Updates Are Necessary
First, understand that, yes, your regression suite will undergo changes as time goes by. Your application will go through changes that will require some of your test cases to also change. Often, changes in the application will render some tests irrelevant, and you’ll have to delete them. And, of course, you’ll need to add new tests to your suite constantly.
Maintenance Must Not Be a Burden
However, if the maintenance of your tests becomes too heavy of a burden, that’s an issue. Here are a few tips to avoid that:
- When doing unit testing prefer writing state-based tests rather than interaction-based ones. Interaction tests tend to result in tests that are too coupled to implementation details and thus break often when said details change.
- Regarding end-to-end testing, avoid tools that use a simplistic approach for locators. These tests fail easily when you change something on the UI. Tools like Testim use AI-based smart locators that are able to identify and retrieve elements from the DOM even if you change ids, CSS classes and the like.
- When starting out with automated testing, prefer a risk-based approach. Coverage is valuable but you won’t achieve it on day one. Instead, focus on identify and testing the riskiest areas in the code base first. That will ensure you get the highest bang for your buck when it comes to your test suite.
Automated Regression Testing: Put It to Work for Your Team ASAP
Software development is a creative endeavor like no other, but the path to reaping its rewards is full of risks. One of the dangers when writing code is to break existing functionality while adding new ones. Even when fixing bugs, we can make older ones—that we thought were dead—come back to life. Each small and seemingly insignificant change poses the risk of a dreaded regression.
If you want to get rid of regressions—and you should want to, believe me—then a suite of regression tests is the solution you seek.
Author bio: This post was written by Carlos Schults.Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.
