NEW RELEASE: TestOps Dashboard—Prioritize your work with data-driven insights. Learn More

How to Put Quality in the Build With Jenkins Test Automation

You probably already have Jenkins running to automate the build process of your applications. However, you might be aware that…

By Testim,

You probably already have Jenkins running to automate the build process of your applications. However, you might be aware that to increase quality in your build process, you need to include testing. Automation without testing is similar to driving a Ferrari without first checking if the brakes work. You’ll go fast, but you might end up crashing as soon as you find the first turn. You already have Jenkins, so why not use Jenkins for test automation?

Testing plays a significant role in today’s world, where everyone is automating as much as possible. So, in this post, I’ll cover why automated testing is so crucial in continuous integration (CI) pipelines. Then, I’ll explain how to integrate testing into a Jenkins job. Finally, I’ll show an example of how to integrate Testim with Jenkins.

Jenkins Test Automation and CI, Explained

Before you roll up your sleeves, let’s take a step back to ensure we’re all on the same page regarding some definitions. Don’t worry, though: if you’re familiar with the terms we’ll be covering, feel free to skip this section and go straight to the hands-on parts of the post.

What Is Continuous Integration?

Continuous Integration, or simply CI, is a fundamental practice for modern software development. It consists on having developers integrating their code to a common repository at least once a day—or, at least, that’s what they’d say when the practice was starting to become widespread, some 20-ish years ago. Nowadays, it’s common for seasoned development teams to integrate their code many times a day.

Why is CI beneficial for software development teams? For starters, it solves the problem of “integration hell”—i.e., developers would work on isolation for weeks or even months and only them attempt to integrate their work. This made integration incredibly hard; there were plenty of merge and logical conflicts. As a result, engineers dreaded integration, postponing it until the last possible moment—which only made the problem worse.

CI solves this by adopting the maxim “if it hurts, do it more often.” Since integration is painful, let’s do it more and more often. A team that integrates every day, for instance, will have less work to do than if they integrated once per week. If you integrate very often, integration becomes virtually trivial.

Expand Your Test Coverage

Fast and flexible authoring of AI-powered end-to-end tests — built for scale.
Start Testing Free

The other major benefit of CI are the checks and validations that are performed with each commit or check-in, which can detect problems with the code way before it makes it into production. Such checks, at the bare minimum, include building the application and running the suite of automated tests. Tools such as static analysis tools or linters also make for great additions to a CI pipeline.

But since this post is about automated testing, that’s what we’ll expand upon in the next section.

What’s the Role of Automated Tests in CI?

As you’ve seen, CI is a practice where developers regularly integrate their code into a shared repository. And to make sure that the integration is successful, there’s a set of automated tests to assess the code’s integrity. Then, if there’s a problem, developers detect and fix it quickly. Jez Humble, co-author of the book Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation, created a test to certify that a team is genuinely doing CI. In Humble’s test, one of the questions asks if an automated suite of tests runs after every code commit. Hence, automatically running tests is one of the key tenets of CI.

The idea of practicing CI is that everyone in the team has the latest working version of the code. That’s why you integrate several times a day. Also, when you include automated tests in your building pipeline, developers’ productivity increases. Developers can push code changes with more confidence because the automated tests are validating their changes automatically. However, tests need to run as fast as possible to provide feedback promptly. Otherwise, it could take too long to fix and find problems, decreasing code quality.

It’s also important to note that CI doesn’t promote the use of any specific tool because what matters most is the practices and culture itself. But, to get a better idea of how to practice automated testing in a CI pipeline, let’s take a look at how you can accomplish it using Jenkins.

Jenkins 101: Definition, Pros and Cons

Let’s now briefly cover Jenkins, explaining what it is, why would you want to use it, and even some of its limitations.

What Is Jenkins?

Jenkins is an open source CI/CD tool. More specifically, it’s an automation server. You can use the power it gives you to automate tasks necessary to build, test, and even deploy your applications.

Jenkins is one of the most popular tools in its category. Much of this popularity is due to its extensibility; thanks to its plugin architecture, Jenkins can be made to work with a large number of tools and services. Users can leverage the existing plugin ecosystem or even create one of their own.

Why Would You Want to Use Jenkins?

As you’ve seen, Jenkins extensible architecture is responsible for much of its popularity. But that’s not all. Here’s a non-exhaustive list of Jenkins advantages:

  • Quick setup: easy installation and configuration.
  • Flexibility: use it just as a CI server or as a complete continuous deployment solution for your app.
  • Free and open source: it suits your pocket.
  • Large community: you won’t need to search for help and support for much time.

Some Limitations of Jenkins in Test Automation

There’s no perfect tool, and Jenkins is no exception. So, here are a few disadvantages of this tool:

  • Redundancy of plugins. Having a large plugin ecosystem is great, but it can sometimes be overwhelming, especially for beginners, when you have multiple plugins for the same end.
  • An UI which could use some improvements. The UI can feel non-intuitive and confusing, especially for beginners.
  • The user management features are a little underwhelming. Some more advanced functions are hard or impossible do to.
  • Low quality of some plugins. The plugins very widely in their quality, and many of them are semi or fully abandoned.

How to Integrate Test Automation in Jenkins

As you might already know, Jenkins works with a plugin system that allows you to build jobs easily. You can install tools and integrations with third-party services like cloud providers. However, let’s not forget that Jenkins is merely another layer on top of the OS the build agent is running. This means that anything you could run in the command line of the build agent can be run through Jenkins. In this case, Jenkins acts as an orchestrator of the commands you need to run in a pipeline.

When you’re new to Jenkins, the easiest way to get started is by using the shell plugin. This plugin allows you to define the command your pipeline wants to run. For instance, a typical set of commands for an application that implements CI is like the following:

  • Compile the application.
  • Run a set of tests, which could be unit, integration, or functional tests.
  • Generate an application artifact that you can deploy across different environments.

For an application written in .NET, the commands would be the following:

If any of the above commands fail, you could use a Jenkins plugin to notify the developer who pushed the code. Also, the team will know that the build is broken because it failed with the latest code integration.

As I said before, depending on your technology stack, there might be plugins where you simply define a set of parameters, such as source location or language version. But in general terms, the workflow would be similar to the one above. You can include other types of tests like integration or functional tests, linting, or code coverage—it depends on the environment you’re targeting. For instance, some folks would recommend that you only run unit tests for a development environment. But, honestly, I’d suggest you run some critical functional tests as well to catch problems early.

Let’s see how to put in practice what I’ve explained so far.

A Concrete Example in Jenkins

The following guide uses Testim to run automated tests using a custom integration in Jenkins. If you don’t have a Testim account, you can sign up for free and record your first test. As I mentioned earlier, you can include other types of tests, like unit tests. Keep automation and parallelization in mind so that the test phase of the pipeline doesn’t take too much time. So, let’s create a simple job in Jenkins and integrate some tests into it.

Prerequisites

Before you start following this guide, you’ll need to have access to the following resources:

  • A Jenkins instance where you can create jobs.
  • Node.js v10 or higher in the Jenkins agent you’ll use to run the job.
  • A Testim account with a test available to run remotely from Jenkins.

Creating a Build Job to Run a Testim Test

Let’s start by creating a Jenkins job. In the left panel, click on the New Item link:

Jenkins screenshot

Enter a name (1) for the job and choose the Freestyle project option (2). Ideally, you should use a Pipeline project and use Git as a source of the pipeline definition. But for now, we’ll use the simplest method, which is the freestyle option. After selecting Freestyle project, scroll down a little bit and click on the OK button:

In the new screen, scroll down to the bottom of the page. In the Build section, click on the Add build step button (1) and choose the Execute shell option (2) if you’re using Linux or Execute Windows batch command if you’re using Windows.

Now, in the Command text area (1), enter the commands (2) that you need Jenkins to run. Let’s continue with the .NET app sample, like this:

Here’s what that code will look like in Jenkins:

Notice that I include the build command before running any test because the first “test” should be that the code compiles. Then, I include the command to run the unit tests. And finally, I add the command to run functional tests with Testim. Also, notice that I include other commands to make sure that the proper npm packages and the Testim CLI are available. If they’re not available, Jenkins installs them.

Lastly, to save the work, click on the Save button. And setting up Jenkins for test automation is as easy as that! Now you have a build pipeline in Jenkins that not only compiles the application, but it also runs unit and functional tests to increase quality.

Increase Your Build Confidence

As you saw, integrating automated testing into your build pipeline is not a difficult task, and testing is crucial when you want to do CI. Keep in mind that having a suite of automated tests you can run after every code commit is one of the key tenets of CI. Moreover, these tests should run fast and cover functional scenarios from your users. If you decide to use Testim, you can record a functional test with a Chrome extension and run that test from a Jenkins job.

Using Jenkins for test automation isn’t your only choice. Jenkins is only one of the many tools out there for doing CI. You can integrate automated testing with other tools like CircleCI, Codeship, or GitLab as well. Check our guides on how to do it. But in general terms, you simply generate the integration code and paste it into your CI tool—as we did today with Jenkins.

Having a CI tool in place is essential to improve the quality of your application. By having every commit or check in trigger a build and test execution, you’ll reap the benefits of having a great test coverage. You’ll be able to properly implement a shift-left testing strategy, detecting and fixing problems as early as possible.

What to read next

Continuous Development Evolves DevOps to DevTestOps

Testim's latest articles, right in your inbox.

From our latest feature releases, to the way it impacts the businesses of our clients, follow the evolution of our product

Blog Subscribe