Puppeteer, Selenium, Playwright, Cypress – how to choose?
At Testim, a lot of what we do is build AI-based features on top of automation frameworks to add stability,…
By Benjamin Gruenbaum,
Share on
At Testim, a lot of what we do is build AI-based features on top of automation frameworks to add stability, accelerate test creation, and improve root-cause analysis.
We evaluate a number of test automation frameworks to understand their strengths and weaknesses, feature differences, and non-functional attributes.
As a new framework, Playwright reaches stable versions, it’s time to compare the popular alternatives.
I believe that we are uniquely qualified to make an evaluation because:
We get to see hundreds of different test automation projects at Testim.
Since we build on top of these tools (and love them) rather than compete with them – we are relatively unbiased.
We have a lot of experience with large-scale projects as well as small scale automation projects. We’ve had companies grow with us from 10 employees to 1000 so we have a pretty good understanding of their struggles.
Disclaimer: We are two people writing this blog post, we are biased because we work a lot on infrastructure. Your experience will probably be different from ours and your mileage might vary. You should choose the best alternative given the criteria of your specific automation project.
Weight the categories to see your scores.
Here is the summary of our ratings. If you want to learn more, then read on. However, since we are all busy and impatient :-), we are giving you the table upfront.
Customize the scores by adjusting the weight to reflect your preference (from very important to not important)
When you are done adjusting the weights, press “calculate score” to see the weighted average scores.
Now back to the blog, to see why we rated things as we did.
Sad realities about E2E tests
Based on survey results and similar to last year – most companies do not have automated end-to-end tests running as part of their CI process. We are at a point where most companies now have unit tests (yay!) but few have automated end-to-end tests.
Most companies we polled (85% out of 284) perform manual end-to-end tests as part of their release process. This makes releasing software significantly slower and more error-prone. Out of the remaining 15%, the vast majority of users running E2E tests use Selenium.
I believe that a big part of it is because in theory software is this nice clean thing and in practice software is a mess.
Most test automation projects like most software projects fail. This is frustrating and we have to change this. Luckily – that’s the trend and the explosion of innovation in the testing space is impacting the way we’re all writing code.
Automation Frameworks
Let’s start with the basics. An automation framework in this post is something that automates your browser by simulating user actions like clicks.
There are two primary ways automation frameworks perform clicks and user actions:
They can use the devtools protocol (or a similar protocol in non-chromium browsers) to execute browser commands “natively” with a privileged capability.
To make a long story short, most frameworks, including Selenium, used to take the first approach – but since it was inherently flakey and problematic – they moved to the second approach.
There are quite a lot of things that happen when you perform a click. The original debugger click eventually ends up as a native operating system call:
Popular Automation Frameworks
There are four popular automation frameworks we get asked about that we’ve evaluated as infrastructure for our AI-based features like smart locators.
We will first discuss them individually and then perform a detailed comparison. We’re going to start with Selenium – it is by far the most popular framework and it’s also the most mature.
Selenium
Uses an HTTP REST JSON protocol for sending commands called the “WebDriver Protocol”. WebDriver is an open standard:
This means that with Selenium it is very easy to use any source language and any target platform. Selenium can automate a vast number of browsers including Internet Explorer, mobile browsers, and even mobile apps (by using Appium).
Since Selenium is a REST JSON API it is pretty easy to understand. Creating a session is just sending a POST request to /session. Performing a click is just sending a POST request to /session/:session-id/:element-id/click.
Underneath the hood – the actual automation is performed by ChromeDriver (in Chrome) which is just an http server. When ChromeDriver starts, it connects via the debugger to Chrome. Then, when the user performs a click it controls the debugger and performs a sequence of “mouse moved, mouse down, mouse up” (using the debugger command Input.dispatchMouseEvent).
WebDriver is also an open standard, so there are a lot of grid options and different ways to scale Selenium to run hundreds or thousands of tests concurrently.
This means that selenium avoids the pitfall of JavaScript event-based automation. Selenium thus has a pretty simple architecture:
Using Selenium + syntax
I would start by installing the official driver (there are some decent alternatives ):
npm install selenium-webdriver
Creating a driver and using it is easy—the syntax is verbose compared to the alternatives but still pretty straightforward:
Executive summary – Selenium
Pros:
Runs on all browsers
Many drivers and clients
Dispatches clicks with a debugger
Lots of grid options
Can use any language like Java or Python and not just JavaScript
Cons:
Not Bi-Directional yet because it’s an Http server. This means things like collecting network events or console logs is very hard
Cypress is an E2E testing framework. It focuses on trying to provide a good developer experience and an integrated environment. Cypress is open-source but it is not based on open standards like WebDriver.
When we evaluated Cypress for internal use – there were a few show-stoppers for us. Clicking in Cypress works like Selenium 1 (the predecessor to Selenium WebDriver) and dispatches DOM Events Directly.
In addition, the lack of support of multiple-tabs and frames and the absence of wait-fors in frames were also problems for us. In practice, our Cypress suite was a lot less stable than the other three alternatives.
Moreover, as a maintainer of an open-source library used by Cypress, I was inclined to like them. But like code in 2013, Cypress doesn’t let you write regular JavaScript. That felt incredibly outdated IMO, even compared to the official Selenium driver.
That said, when evaluating Cypress we enjoyed the great documentation and the streamlined process.
Executive summary – Cypress
Pros:
Good documentation
Friendly community
Uses libraries Benji was involved with under the hood ❤️
Cons:
Uses the same techniques Selenium 1 switched away from for automation
No multi-tab support
Tests with multiple frames were very flakey when we evaluated them
Very limited browser support
The jQuery-based API felt very outdated
In order to do parallelism well, you need to use vendor-locked software.
Puppeteer
Puppeteer is a popular test automation tool maintained by Google. It automates Chrome and Firefox. It is relatively simple and stable.
Fundamentally Puppeteer is an automation tool and not a test tool. This means it is incredibly popular for use cases such as scraping, generating PDFs, etc.
Puppeteer uses the same debugger protocol Selenium (well, ChromeDriver) uses to perform clicks and in practice Puppeteer (Playwright which we’ll discuss later) and Selenium, all use the same code for performing clicks. In practice, Puppeteer’s architecture looks something like this:
So, it’s just a really simple and cool wrapper over the devtools protocol (and the equivalent for Firefox).
Puppeteer also takes care of downloading Chrome for you and is generally easier to set up than Selenium for the development flow.
Using Puppeteer + syntax
npm install puppeteer
Then the syntax is pretty straightforward, modern and nice:
Puppeteer also gives you direct access to the CDP if you need it. Which can be very useful at times and in general it feels like there are fewer moving parts.
Executive summary – Puppeteer
Pros:
Simple to set up
Good documentation
Installs Chrome in a working version automatically
Thin wrapper
Bi-Directional (events) – automating things like console logs is easy
Maintained by Google.
JavaScript first, so the code feels very natural
Cons:
Limited cross-browser support—only Chrome and Firefox
Feels like an automation framework and not a test framework—you often have to re-implement testing-related tools
Grids (running concurrently) in production are often a challenge
The automatic browser set up downloads Chromium and not Chrome and there are subtle differences between the two.
Playwright
Playwright is the new kid on the block. It is written by some of the same people who authored Puppeteer and it is maintained by Microsoft.
Playwright utilizes the same architecture as Puppeteer and is a thin WebSocket client. It uses a very similar syntax and language but there are a few differences—namely that Playwright supports more browsers (Safari) and that Playwright feels like a test automation tool rather than just an automation tool.
This means there are things that are easy to do with Playwright that are harder with Puppeteer:
Selecting an element by text instead of by a CSS selector
Waiting for elements to be available automatically
Solid network validations and network mocking.
Those things are all possible with Puppeteer but feel natural with Playwright. Playwright still feels like infrastructure to build on, but it feels like test infrastructure and not automation infrastructure. They are also working on isolated sessions in browsers for grids which I’m not entirely a fan of but it’s definitely interesting. The syntax and installation are so similar to Puppeteer there is no need to copy/paste it again just to show it.
Executive summary – Playwright
Pros:
Simple to set up
Test framework stability features. When we evaluated Playwright compared to Cypress internally, Playwright consistently outperformed Cypress in terms of stability
Installs Chrome, Firefox or WebKit (Safari) in a working version automatically
Thin wrapper
Bidirectional (events) – automating things like console logs is easy
Maintained by Microsoft people with experience maintaining Puppeteer
JavaScript first, so the code feels very natural
Cons:
Very new so the APIs are evolving
No support for IE11 or non-browser platforms
Still very few integrations and tutorials
Grids are still a challenge
Docs and community are not as good as the others.
Key Differences
Disclaimer: This is just our PoV. It’s built based on evaluation criteria we believe are important, but biased towards tools that Testim can use as infrastructure. Your mileage, criterion, and cats might vary.
Cypress: ❌ (If you want us to add support for Cypress in the playground please let us know)
Puppeteer: ✅ Yes (with Testim Playground)
Playwright: ✅ Yes (with Testim Playground)
Note: when we polled companies – test creation speed and in particular, the creation speed of stable tests was a severely limiting factor in the success of automation projects. When evaluating tests authored with Testim that was not the case.
Trusted Actions
This criterion means dispatching events by the user agent which allows for user agent behaviors like hovers.
Selenium: ✅ Yes
Cypress: ❌ No support (can use Puppeteer plugin)
Puppeteer: ✅ Yes
Playwright: ✅ Yes
Parallelism Grids and Infrastructure
Selenium: ✅ Yes (managed, costly) or build your own solution
Cypress: 🤷 Only in their closed source paid cloud or build your own
Puppeteer: ❌ Usually people build their own (will change soon)
Playwright: ❌ Usually people build their own (will change soon)
Performance
End-to-end tests are very fast in practice but people suffer from misconceptions regarding the execution speed of Selenium tests. Typically, it’s the website or web-app that are slow and the tests end up waiting for the web app to be ready most of the time.
Selenium: ✅ It’s fast enough, really
Cypress: ✅ It’s fast enough, really
Puppeteer: ✅ It’s fast enough, really
Playwright: ✅ It’s fast enough, really
Stability
This means how often tests fail after being authored, other than when detecting a real application bug
Selenium: ❌✅ Complex Automatic Wait For mechanism
Cypress: ❌✅ Complex mechanism that doesn’t work with frames
Puppeteer:❌✅ Wait fors for certain things, but have to waitFor manually for others
Playwright: ❌✅✅ Better wait fors for certain things, but have to waitFor manually for others
In practice, waiting for elements in test infrastructure really depends on how your website or app works.
Smart(er) Locators
Selenium: ❌ No support for selecting elements in multiple ways
Cypress: ❌ No support for selecting elements in multiple ways
Puppeteer❌ No support for selecting elements in multiple ways
Playwright: ❌✅✅ Very promising start of supporting custom selector engines.
Self-Healing tests and automatically improving tests
Selenium: ❌ No
Cypress: ❌ No
Puppeteer❌ No
Playwright: ❌ No
If you are not sure what self-healing tests are check out this webinar we did.
Debugging
Selenium: ❌✅ A bit hard to figure out all the terminology. Debugging remote grids relies on the grid provider
Cypress: ❌✅ You’re not even writing regular JavaScript, you’re chaining promises. Makes up with DOMs
Puppeteer: ✅ Writing and debugging JavaScript from your IDE
Playwright: ✅ Writing and debugging JavaScript from your IDE
When we built TDK we took the same approach as Playwright and Puppeteer, and we think it’s the preferable one.
Documentation and Resources
Selenium: ✅✅ Very large community. Many testers.
Cypress: ✅✅ Small community but buzz – and very nice documentation.
Puppeteer: ✅ Small community but lots of tutorials at this point
Playwright: ✅❌Docs and tutorials out of date due to changing API. Still feels a bit experimental.
Note that this is a guide for evaluating test infrastructure. No matter what you choose, unless you go with a managed platform you will need to spend a considerable amount of time on your test infrastructure (unsurprisingly and like any other software development project).
The biggest mistake in test automation projects we see is people don’t plan. They start writing tests and then abandon the project when it becomes unmaintainable. Like we stated earlier, most test automation projects fail and most companies perform manual QA.
If you don’t want to use JavaScript you are probably better off using Selenium anyway. The community and ecosystem size for Java and Python support are considerably smaller for all frameworks outside of Selenium. There are projects like jpuppeteer and puppeteer-sharp but they are third-party and much much smaller than the official Selenium alternatives.
If you want to use JavaScript then you can either mix Selenium and Puppeteer or use Playwright. You can’t mix Playwright and Selenium together at the moment.
Testim also created an open-source project called Root Cause to help Puppeteer and Playwright users troubleshoot their tests. We also created a couple of free tools that allow you to record a test and export code for Puppeteer or Playwright.
All three are good choices.
Just remember that writing a successful automation project is more than just infrastructure. Treat automation like any other software project. You wouldn’t write unmaintainable frontend code (intentionally :])—don’t write tests or code you can’t maintain.
Automation is exploding with new things. You can and probably should probably get Involved!
These tools are all open source. Get involved and you can help make next year’s comparison ✅ all around. Automation has a lot of tradeoffs. Test automation tools are different from each other, because of when they were developed, who developed them, and what goals they set out. There is no, one-size-fits-all automation framework, and many companies mix and match depending on their applications and needs.
There are also great commercial options, like Testim. Testim simplifies test automation by turning recorded user flows into test steps that can be configured, customized, or exported as code. We integrate with your development pipelines for event-triggered test runs, and we manage the infrastructure so you don’t have to. Start by getting a free account and try it yourself.
Test Data Is Critical: How to Best Generate, Manage, and Use It
When it comes to testing, data is king! In an increasingly digitized world, data is becoming more important for many…
QA, Selenium
How to Configure and Run Selenium in Jenkins
Setting up a Selenium-Jenkins task comes with a lot of advantages for developers. Especially given how continuous integration / continuous…
CI/CD
How to Implement Record and Playback in Selenium
We're back with another testing-related subject. Today's topic is pretty straightforward: We'll show you how to implement record and playback…
Selenium
Testim's latest articles, right in your inbox.
From our latest feature releases, to the way it impacts the businesses of our clients, follow the evolution of our product
We and selected partners, use cookies or similar technologies to provide our services, to personalize content and ads, to provide social media features and to analyze our traffic, both on this website and through other media, as further detailed in our cookie policy
This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.