Let’s talk about automated testing tools. Are they valuable? While some people seem to think of them as a mere accessory, that couldn’t be further from the truth.
An old teacher of mine used to say that one year in the IT world was actually worth ten years—such was the pace of innovation and progress. After graduating and becoming a software developer, I could see for myself that he was right. To remain competitive in such a fast-evolving landscape, companies worldwide must make use of automation, which includes having a sound automated testing strategy in place.
And to design, implement, and maintain said strategy, you’ve got to have the right tools in your toolbelt. Sure, tools aren’t everything. Like in any field, mastering the fundamentals is always going to be vital. As soon as you get the basics out of the way, finding and employing the best tools available becomes imperative. And that’s what this post is all about: helping you on the “finding the best tools” part.
By the end of this post, you should have a mature understanding of the current automated testing tools scene. You’ll then be able to make an educated decision about which tool or tools are the best fit for your needs. Let’s get started!
The 11 Automated Testing Tools You Should Know About
We’re now covering eleven automated testing tools that you need to be aware of. Keep in mind that our list will be a little bit different from other lists you see on similar named posts around the web.
Other posts about automated testing tools tend to focus almost exclusively on UI testing tools. This list, on the other hand, presents more variety. We’ll cover tools that fall into several different automated testing categories—unit testing, acceptance testing, and yes, UI testing.
When you’re done with this article, you should have a good understanding of the available offerings out there to help you with each step of your automated testing strategy.
The xUnit Family of Unit Test Frameworks
Back in 1999, Kent Beck released a unit test framework for Smalltalk. It was called SmalltalkUnit (or simply SUnit), and its architecture-inspired an array of frameworks for unit testing on a variety of languages and platforms.
After SUnit, the first framework to appear was JUnit, which was a port of SUnit to the Java language. Since then, many others have followed. For instance:
Unit testing is one of the most fundamental types of automated tests, so a xUnit test framework is a tool that deserves a place in your toolbox.
- Unit testing is one of the most important types of automated testing.
- To perform unit testing, you might use some of the many unit testing frameworks available, based on the xUnit family.
- Tools in the xUnit family are free and open-source.
PIT Mutation Testing
PIT is a mutation testing framework for Java. That’s as straightforward a definition as it can get. But the definition is useless if you don’t know what “mutation testing” means.
Mutation testing is the process of deliberately adding defects to your codes. Those faults are called “mutations.” After the framework introduces a mutation, it runs your unit tests. If they fail, we say the mutation was killed. Otherwise, the mutation survived.
The “mutations killed/mutations survived” ratio you get is a reliable indicator of your unit tests suite’s quality. Think about it. If all of your tests still pass after you damaged a portion of the production code, that can only mean one of two things—either your current tests are wrong, or there aren’t enough tests.
You could do all of this by hand, sure. But it would be a prolonged, tedious, and error-prone process.
PIT website: pitest.org
- You could ensure your unit tests work by manually introducing defects in the production code.
- Mutation testing helps you automate that process.
- PIT is the state-of-the-art mutation testing tool for Java.
Stryker website: stryker-mutator.io/stryker
- Mutation testing is available for many programming languages.
Cucumber is an interesting item on our list since it wasn’t designed to be an automation testing tool from the start. Instead, Cucumber helps teams—including non-technical people—write specifications and use cases for software features in a clear, unambiguous way.
Cucumber allows teams to write software using the BDD (behavior-driven development) approach by allowing technical and non-technical members to describe software use cases using a common language.
After you create one or more scenarios for the feature you wish to document, Cucumber allows you to generate automated tests from them. You can do that using test automation libraries that your language already supports. In Java, for instance, you could use JUnit.
Keep in mind that the tests you end up with by employing Cucumber are not unit tests. These are higher-level tests—or so-called “executable specifications.” They describe the way each software feature is meant to behave from the user’s perspective. Such tests verify a functionality from the UI through the database, passing through every layer.
Unit tests, on the other hand, verify that a single unit works in complete isolation. It gives the developer confidence that their code does what they think it does. But it doesn’t check how the units integrate or how the different layers work in harmony to fulfill the user’s needs.
In other words, Cucumber doesn’t compete with the xUnit style framework; rather, it complements them.
Cucumber website: cucumber.io
- BDD (behavior-driven development) is a methodology for developing software in which use cases are turned into executable tests.
- BDD’s goal is to foster collaboration between technical and non-technical people inside organizations.
- Cucumber is one of the most popular tools for performing BDD.
Selenium is another interesting case in that it is not, per se, an automated testing tool. Instead, it’s a browser automation tool. With it, you can automate interactions with a web browser. What you do with this capability is up to you.
In practice, what most people use Selenium for is indeed testing. Using the tool, they automate interactions with their web applications, effectively creating automated tests for their web UIs.
Selenium—without the aid of Selenium IDE—isn’t a standalone tool. It’s more like an API, against which you can program—using your favorite language or platform—to create the browser automation that you need.
Selenium website: www.selenium.dev
- Selenium is a popular web automation tool.
- Developers can use Selenium’s API along with their favorite programming language to drive browsers to perform the needed tasks.
- Though not exclusively a testing tool, Selenium is often used to that end.
Serenity is a Java framework that operates with BDD tools (e.g., Cucumber) while acting as a wrapper on top of Selenium WebDriver.This tool makes it easier to write BDD-style tests and even Selenium tests. It frees you from writing a lot of boilerplate code. It also comes with useful built-in features, allowing you to take screenshots, easily integrate with Jira, parallelize tests, and much more.
With Serenity, you can create highly detailed reports. The tool can create documentation that you can use both as test results and documentation for your app.
Serenity website: serenity-bdd.github.io
- Serenity BDD is an open-source tool that aims to produce living documentation.
- It easily integrates with BDD tools such as Cucumber.
- You can use Serenity as a wrapper around Selenium, avoiding boilerplate code and simplifying much of the process of using it.
FitNesse is an open-source tool used for writing and verifying acceptance criteria for an application. It’s an acceptance-testing framework. FitNesse is a wiki server, allowing its users to express test scenarios as tables containing input and output examples.
Since it’s a wiki, Fitnesse tables are easy to create and edit. They can and should be edited by stakeholders and other business people in the organization. Then, developers write “fixtures”—that is, the underlying testing code—that gets executed when someone tries to run the tests in the tables. The idea is that the whole team—coders and non-coders—collaborate with the wiki, writing, and testing requirements, to create a shared understanding of the application.
FitNesse website: docs.fitnesse.org/FrontPage
- FitNesse is a wiki server users can use to express specifications as tables.
- It enables and encourages the participation of stakeholders and business people in the testing creation process.
- FitNesse is also an open-source tool.
With such tests, you can verify that your API is working as intended. Also, you can use them to avoid regressions. That is, determining that new changes to the API haven’t introduced bugs to previously working features.
Postman website: postman.com
- Postman allows people to test calls to APIs.
- It enables users to group common calls into collections they can share with colleagues.
- Postman allows you to create automated test suites.
Since unit tests aren’t supposed to interact with external dependencies, you need to replace them with fake implementations. That’s where a mocking tool comes in handy, and a popular option for the JS world is Sinon.
- Jasmine website: jasmine.github.io
- Karma website: karma-runner.github.io
- Jest website: jestjs.io
- Sinon website: sinonjs.org
Performance Testing Tools
Many people think of performance as being a non-functional requirement, though I beg to differ. I prefer to side with Jeff Atwood, who’s been saying that performance is a feature for almost a decade by now. If performance is so vital, it makes sense that you should test it somehow.
So, this section is about a category rather than a specific tool. Namely, tools that help you assess the performance of your application.
We’ll start with Artillery, which is a load testing framework. Load testing is a specific type of performance testing in which we verify how the system behaves when receiving a high number of requests simultaneously. In other words, we want to see how the system reacts under peak load conditions.
JMeter is also an open-source tool used for load and performance testing. Maintained by the Apache Foundation, JMeter is a powerful and complex tool. It supports a wide variety of protocols but can be overblown if you only have basic load testing needs.
An important difference between JMeter and Artillery is that while the former is GUI-oriented, the latter is script-oriented, so that’s another criterium you must factor in when making your decision.
- Performance is essential for web applications, and you should test it.
- Artillery is a lightweight load testing tool for testing backends and APIs.
- JMeter is another load testing tool. It’s compelling, though more complex.
Finally, take a look at Testim.
Testim offers a straightforward recording function, allowing non-developers to create UI tests easily. It also enables testers to report a bug and automatically generate a test that reproduces it. As already covered on this blog, similar tools in the past tended to result in fragile tests. Testim is different since its greatest asset is artificial intelligence.
It uses AI to improve the authoring and execution of tests. How does it do that? It analyzes the DOM objects of the page under test, extracting them along with their properties.
Then, based on this analysis, the AI figures out the optimum strategy to locate a given element. What that implies is as simple as it is powerful—even when a developer changes some attribute of the element, the test still works.
This “dynamic locator” approach is the primary advantage of using Testim compared to other frameworks— authoring and executing tests becomes way faster and stable.
There Are Plenty of Automated Testing Tools. How To Choose the Right One for Your Needs?
Due to the sheer number of existing automated testing tools, it’s doubtful that you’ll get to use—or even know—most of them. So, it’s not surprising when developers, QA professionals, and other people involved with software testing feel overwhelmed when faced with the daunting task of deciding which automated testing tools their teams and organizations should adopt.
It is overwhelming.
So…what tools should you choose?
Well, at the end of the day, no one knows more about the needs of your team or company than you. But there are easy, practical questions you can think about to guide you towards a better decision. Here they are.
Which Types of Software Do You Create?
It’s not only automated testing tools that come in different shapes and sizes. The software itself comes in many different flavors: you can have hardware drivers, operating systems, embedded system software, mobile applications, desktop applications, web apps, RESTful APIs, and that’s just to name a few. The type of software your organization creates might help you decided which types of software testing—and software testing tools, by consequence—are the best match for it.
For instance, if you develop software that doesn’t have a GU (graphical user interface), you don’t need to worry about GUI testing and the tools that enable it. There are types of software that have no UI, graphical, or otherwise. If that’s the type of software you create, you don’t need any form of UI testing at all.
There are also types of automated tests that are universal or close to that. Unit testing certainly fits the bill: it’s hard to find a scenario where it’s not useful.
What’s Your Ratio of Manual vs. Automated Testing?
This post is all about automated testing—as are most posts you’ll find on this blog. That doesn’t mean manual testing is dead.
Quite the contrary. Most solid software testing strategies still have a place for manual testing. It’s all a matter of determining what the right balance is between the manual and automated approaches.
When you find that balance, it will help you in a lot of test-related decisions, among which is the decision of which testing tools to invest in. Let’s say your organization is starting to adopt automated testing by first tackling the base of the testing pyramid. In that case, it makes sense to focus most of your tool investments on the unit testing layer.
Do the People Who Perform Software Testing Have Coding Skills?
Here’s another key question to answer when determining the automated tools that make the most sense for your organization: who performs testing in your organization?
I’ll follow that question with another one: can they code?
These are important questions to ask because despite the lines between testers and developers becoming more and more blurred as time goes on, they’re not completely blurred yet. You’re likely to have testers with no coding skills. And if that’s the case, it makes sense for your organization to invest in codeless testing tools, so you can leverage the experience and knowledge of testing professionals who are not developers.
What’s Your Budget?
You know that thing they say about free lunches, right? Yes, exactly: they don’t exist. Testing tools have a cost. The size of your budget will inform—or, put it more bluntly, limit—the available pool of tools you’re able to pick from.
Keep in mind that what we mean by “cost” goes well beyond just the price tag. Even tools you can obtain for free—such as the open-source ones we’ve mentioned in this post—have some fort of cost attached to them. This cost might assume the form of a steep learning curve, for instance. Or maybe installing and setting up the tool is difficult, and the existing documentation is of little or no help at all. The organization might even need to hire specialized training to capacitate its employees to use the tool.
In short: don’t forget to consider the total costs of ownership.
Automated Testing Tools in All Shapes and Sizes: Know Them So You Can Master Them
It took us long enough, but we—as in the software industry—are now finally employing automation to improve our own work at all phases of the process. One of the manifestations of this trend is the modern automated testing movement. For a modern software development shop, having a mature automated testing strategy is no longer a luxury. It’s essential.
And a vital step to creating such a strategy is to know and master the available tools. Today’s post helped you become aware of some of the tools at your disposal. You can also look at our five-step process to learn how to identify your needs’ best automation platform.
The tools we’ve covered today fall into many different categories of automated testing, and this is by design.
You now have a high-level view of the tools available and the types of automated testing they support. The next step is to zoom-in and learn more about the tools and test style that your repertoire is lacking to improve it even more.
Carlos Schults wrote this post. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.