Let’s talk about automated testing tools. Are they valuable? While some people seem to think of them as a mere accessory, that couldn’t be further from the truth.
An old teacher of mine used to say that one year in the IT world was actually worth ten years—such was the pace of innovation and progress. After graduating and becoming a software developer, I could see for myself that he was right. To remain competitive in such a fast-evolving landscape, companies around the world must make use of automation, which includes having a sound automated testing strategy in place.
And to design, implement, and maintain said strategy, you’ve got to have the right tools in your toolbelt. Sure, tools aren’t everything. Like in any field, mastering the fundamentals is always going to be vital. As soon as you get the basics out of the way, though, finding and employing the best tools available becomes imperative. And that’s what this post is all about: helping you on the “finding the best tools” part.
By the end of this post, you should have a mature understanding of the current automated testing tools scene. You’ll then be able to make an educated decision about which tool or tools are the best fit for your needs. Let’s get started!
The 9 Automated Testing Tools You Should Know About
We’re now covering nine automated testing tools that you need to be aware of. Keep in mind that our list is going to be a little bit different than other lists you see on similar named posts around the web.
Other posts about automated testing tools tend to focus almost exclusively on UI testing tools. This list, on the other hand, presents more variety. We’ll cover tools that fall into several different categories of automated testing—unit testing, acceptance testing, and yes, UI testing.
When you’re done with this article, you should have a good understanding of the available offerings out there to help you with each step of your automated testing strategy.
The xUnit Family of Unit Test Frameworks
Back in 1999, Kent Beck released a unit test framework for Smalltalk. It was called SmalltalkUnit (or simply SUnit), and its architecture inspired an array of frameworks for unit testing on a variety of languages and platforms.
After SUnit, the first framework to appear was JUnit, which was a port of SUnit to the Java language. Since then, many others followed. For instance:
- NUnit, for the .NET platform
- PHPUnit, for the PHP language
- PyUnit, for Python
Unit testing is one of the most fundamental types of automated tests, so a xUnit test framework is a tool that deserves a place in your toolbox.
PIT Mutation Testing
PIT is a mutation testing framework for Java. That’s as straightforward a definition as it can get. But the definition is useless if you don’t know what “mutation testing” means.
Mutation testing is the process of deliberately adding defects to your codes. Those faults are called “mutations.” After the framework introduces a mutation, it runs your unit tests. If they fail, we say the mutation was killed. Otherwise, the mutation survived.
The “mutations killed/mutations survived” ratio you get is a reliable indicator of the quality of your unit tests suite. Think about it. If all of your tests still pass after you damaged a portion of the production code, that can only mean one of two things—either your current tests are wrong, or there aren’t enough tests.
You could do all of this by hand, sure. But it would be a prolonged, tedious, and error-prone process.
PIT website: pitest.org
Cucumber is an interesting item on our list since it wasn’t designed to be an automation testing tool from the start. Instead, Cucumber helps teams—including non-technical people—write specifications and use cases for software features, in a clear, unambiguous way.
Cucumber allows teams to write software using the BDD (behavior driven development) approach, by allowing technical and non-technical members of the team to describe software use cases using a common language.
After you create one or more scenarios for the feature you wish to document, Cucumber allows you to generate automated tests from them. You can do that using test automation libraries that your language already supports. In Java, for instance, you could use JUnit.
Keep in mind that the tests you end up with by employing Cucumber are not unit tests. These are higher-level tests—or so-called “executable specifications.” They describe the way each software feature is meant to behave, from the user’s perspective. Such tests verify a functionality from the UI through the database, passing through every layer.
Unit tests, on the other hand, verify that a single unit works in complete isolation. It gives the developer confidence that their code does what they think it does. But it doesn’t check how the units integrate, or how the different layers work in harmony to fulfill the user’s needs.
In other words, Cucumber doesn’t compete with xUnit style framework; rather, it complements them.
Cucumber website: cucumber.io
Selenium is another interesting case in that it is not, per se, an automated testing tool. Instead, it’s a browser automation tool. With it, you can automate interactions with a web browser. What you do with this capability is up to you.
In practice, what most people use Selenium for is indeed testing. By using the tool, they automate interactions with their web applications, effectively creating automated tests for their web UIs.
Selenium—without the aid of Selenium IDE—isn’t a standalone tool. It’s more like an API, against which you can program—using your favorite language or platform—to create the browser automation that you need.
Selenium website: www.selenium.dev
Serenity is a Java framework that operates with BDD tools (e.g., Cucumber) while acting as a wrapper on top of Selenium WebDriver.This tool makes it easier to write BDD-style tests and even Selenium tests. It frees you from writing a lot of boilerplate code and it also comes with useful built-in features, allowing you to take screenshots, easily integrating with Jira, parallelizing tests, and much more.
With Serenity, you can create highly detailed reports. The tool can create documentation that you can use both as test results and documentation for your app.
Serenity website: serenity-bdd.github.io
FitNesse is an open-source tool used for writing and verifying acceptance criteria for an application. That is to say, it’s an acceptance-testing framework. FitNesse is a wiki server, allowing its users to express test scenarios as tables containing input and output examples.
Since it’s a wiki, Fitnesse tables are easy to create and edit. They can and should be edited by stakeholders and other business people in the organization. Then, developers write “fixtures”—that is, the underlying testing code—that gets executed when someone tries to run the tests in the tables. The idea is that the whole team—coders and non-coders alike—collaborate with the wiki, writing and testing requirements, in order to create a shared understanding of the application.
FitNesse website: docs.fitnesse.org/FrontPage
With such tests, you can verify that your API is working as intended. Also, you can use them to avoid regressions. That is, determining that new changes to the API haven’t introduced bugs to previously working features.
Postman website: postman.com
Since unit tests aren’t supposed to interact with external dependencies, you need to replace them with fake implementations. That’s where a mocking tool comes in handy, and a popular option for the JS world is Sinon.
- Jasmine website: jasmine.github.io
- Karma website: karma-runner.github.io
- Jest website: jestjs.io
- Sinon website: sinonjs.org
Finally, take a look at Testim.
Testim offers a straightforward recording function, allowing non-developers to easily create UI tests. It also enables testers to report a bug and automatically generate a test that reproduces it. As already covered on this blog, similar tools in the past tended to result in fragile tests. Testim is different since its greatest asset is artificial intelligence.
It uses AI to improve the authoring and execution of tests. How does it do that? It analyzes the DOM objects of the page under test, extracting them along with their properties.
Then, based on this analysis, the AI figures out the optimum strategy to locate a given element. What that implies is as simple as it is powerful—even when a developer changes some attribute of the element, the test still works.
This “dynamic locator” approach is the primary advantage of using Testim compared to other frameworks— authoring and executing tests becomes way faster and stable.
Automated Testing Tools in All Shapes and Sizes: Know Them so You Can Master Them
It took us long enough but we—as in the software industry—are now finally employing automation to improve our own work, at all phases of the process. One of the manifestations of this trend is the modern automated testing movement. For a modern software development shop, having a mature automated testing strategy is no longer a luxury. It’s essential.
And a vital step in order to create such a strategy is to know and master the available tools. Today’s post helped you become aware of some of the tools at your disposal. You can also take a look at our five-step process to learn how to identify the best automation platform for your needs.
The tools we’ve covered today fall into many different categories of automated testing, and this is by design.
You now have a high-level view of not only the tools available but also the types of automated testing they support. The next step is to zoom-in and to learn more about the tools and test style that your repertoire is lacking, in order to improve it even more.
This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.