The process of building and deploying software is undergoing a revolution, not unlike the invention of the assembly line. That seminal development allowed Ford Motor Company to crank out consumer vehicles so fast, it positioned them as the market leader almost immediately.
Today, old manufacturing processes have been replaced by the knowledge-worker driven software development process. Unfortunately, the same traditional wisdom that applies to Six Sigma style manufacturing cannot be applied to software development. It is an entirely new discipline that has only existed as a major, widespread career for the past 30 years.
In manufacturing, more bodies on the line mean more products out the door, but in the software development world, this has traditionally been an inverse ratio: the more people work on a project, the slower it moves overall. That’s because software development is mental gymnastics. Developers must keep the whole model of the program in their minds while they work, and once things get too big to keep in a single person’s brain, inter-personal communication and coordination become a major bottleneck.
That’s why agile software development was invented. By cutting software into smaller bits, or “Features,” teams can focus on the same task for shorter periods, and ensure new ideas make it into software projects faster.
Agile processes took enterprise development from releasing yearly, or every six months, to releasing once a quarter, or even once a month. That, in turn, drove the business revenues, as new features drew new customers, and software increasingly became a line of business, rather than simply a business tool.
Today, this increasing velocity of software development has led to continuous integration, and continuous deployment: practices that allow the software to be updated every week, or sometimes every day. For those teams with the skills and processes in place to do it properly, they can even ship new code every hour.
That requires a careful balancing act, however. Teams must be disciplined, processes must be optimized, and above all else, tests should be automated and easy to maintain. Specifically, key verifications should be automated and available on demand.
Testing as enabler
Test Automation is the single most important factor in maintaining software quality at high velocity in CI/CD. In order to enable testing at this high rate of release, automation and long-term management of tests are top priorities for any manager to consider. That is to say, they should be at the top of their to-do list, but not a must: sometimes manual testing remains the best choice.
Test automation is easily seen in the wrong light, however. Managers can look at the idea of automation and see it to be a new way to reduce headcount or to save money on overall development costs. This is incorrect, however, and pursuing test automation for these reasons alone is to miss the forest for the trees.
The real reason test automation enables CI/CD is that it shortens the feedback loop between developers and test results. With proper automated tests in place, and running at scale, developers can see the results of their work in minutes, allowing them to quickly return to fix obvious problems.
When a developer runs a build, they are no longer working. Historically, these developers would get coffee, play ping-pong, or even go to lunch. If a build and test process takes an hour, or worse yet, requires an actual person to do the work, your developer is burning cycles when they’re waiting.
If you’re running 20 builds a day at 10 minutes each, that’s 200 minutes wasted. That’s not only going to put your developers behind schedule, it’s also going to hamper their attempts to push changes to production and bolster the company’s bottom line with new products and features.
If that 20 builds-per-day can be reduced to 1 minute each, you’ve just saved every developer 3 hours a day. That means they can cut more code, ship more features, and fix more problems.
Of course, making builds run fast requires tests to run fast. Regression tests are repetitive and must be maintained over time when buttons move, or workflows experience simple changes. They cannot be left out, however, as they can catch showstopper bugs that were previously fixed but became live again due to code updates. Behind the UI, functionality tests are perhaps your most important path to verification: in essence, you should be testing the business itself when testing functionality. If your application’s use case is directly tied to business revenue intake, making sure it’s stable while developing at velocity requires tightly managed tests across the range of the application.
When business value derives directly from the throughput of customers in an application, testing performance and security are also must-haves for every build cycle. Customers can quickly change to a competing product, thanks to the modern business Web, and the never-ending deluge of security concerns online also means that your data must be protected during testing. This adds another layer of complexity to the testing toolchain, as live data must be scrubbed of identifying information before being used in a test environment.
With that many tests required to run on every build that goes to production, and with multiple builds being pushed every day (in the dream scenario), management of those tests must not be cumbersome or difficult.
UI testing, for example, is the most difficult type to get right consistently, and in an automated fashion. If this is where your testing plans begin, you’re building a skyscraper from the top down. Instead, unit testing should be the fundamental, well, unit of your testing regimes. Building up from those foundations, a full battery of tests can be constructed to ensure pushing to production is only done when things are known to be stable.
Every testing environment is different and will require a mix and match of the various levels of tests. From inside the code, to inside a harness around the code, every test is a unique snowflake in your unique project. For this reason, test automation requires careful planning, and a distinct roadmap to build out, based on your team’s needs, infrastructure, and tooling.
Automated testing: 8 strategies and tactical plans
As Donald Rumsfeld once said, “You go to war with the army you have, not the army you wish you had.” That applies in software development, as well. You’re likely working with a selection of tests, already, and they’re likely not as good as you would want.
But you’re going to start there no matter how you feel about those tests. The first step on the path to true CI/CD integrated test automation is to assess your tests. What do you have, and what do you need? Are you short on UI tests? Are there separate mobile tests and Web tests? Is there a backend API you can test directly, without an obfuscating UI layer?
If there is an API layer, that’s a great place to begin automation. You want to find the spaces in the application that ties directly to the business logic and the business use case. The core of the application or service is where the most important functionality resides. Don’t try to automate everything; instead, focus on the most important bits of the application, and automate the tests as close to the core as possible.
Just automating these tests isn’t going to be the whole job, anyway. Once you begin, you’ll also want to ensure the test environment, data, and reporting are also in place and served in a reliable fashion for on-demand testing. The goal here is to get the infrastructure for automated testing at scale into place before you actually try to automate everything.
Once you have these infrastructure components in place to support that small set of core tests, and once that pipeline is running smoothly, you can begin to add in the other tests in your arsenal and automate the as appropriate.
Now that you’re into the meat of test automation, here are some tips and tricks for success.
Make a list, prioritize it, focus on the functionality that converts users, if that’s your most important metric.
Build for reusable parts, with atomic, clear naming conventions that tie back into the vocabulary of your source code, the variables, the classes. Reuse comes from proper architecture, however, not from copying and pasting.
03 TWO LEVELS OF ABSTRACTION, MAYBE MORE
If you have clicked on your tests, you are doing it wrong. Those clicks need to be driven by an automated system that can handle changes in UI and functionality. Otherwise, your teams will spend all of their time updating tests as the target changes. Think about your average code change, or new feature added. If it was a major change to the UI or functionality, it likely broke a lot of tests, too. Automated test management and design can help to alleviate the need to rewrite tests with every passing software update. Use the Page Object pattern: wrap your HTML in an API that will allow you to access elements directly with tests.
04 SINGLE-PURPOSE TESTS
Keep those tests scoped to a single purpose. When things go wrong, the last place you want to be wondering about is the test layer. If your tests test multiple things, they’re single points of failure for multiple points of contact. This is exactly why converting manual test cases to automated tests, one-to-one is a very bad idea.
05 USE A CONSISTENT INITIAL STATE
Variables are bad. This is a science, not a college final, a marathon, not a sprint.
06 NEVER SLEEP A PROCESS
This is the source of all evil. When things don’t wake up, processes stall. Keep all processes awake during test processes. Timeouts are another source of common problems. Keep a close eye on them.
07 MINIMIZE THE NUMBER OF CONDITIONS
The more moving pieces you have in a piece of software, the more things can go wrong. Setting default states for applications in testing environments can be a useful way to minimize variance. Certainly, there is a time for fuzzing and for testing every string length under the sun, but when testing a specific case, keep the moving variables to a minimum and ramp them up as the application proves itself to be stable in simpler tests.
08 PREPARE FOR CHANGE
Some people think automatically that units tests are whitebox, but any time you make changes to your application, you have lots of refactoring to do. Automated testing and automated test management will help you deal with the change. Do not fear it, expect it, prepare for it, get it under control.
Who can do test automation?
Test automation requires software development skills. This is not a pursuit for unskilled workers, as has been thought in the past. The days of teams clicking on screens and recording their mouse movements are over. Test automation is a discipline akin to any other development or administrative role in software.
Testers require software development knowledge, scripting expertise, testing skills, and above all, excellent communication skills. As they’ll be in charge of testing other people’s work, it’s key that they learn a common vocabulary for discussions and the naming of components and requirements.
Having a shared language makes it easy for everyone on the project team to stay in sync. DevOps is all about breaking down the silos between ops, development, and test, and thus, communication lines must flow just as easily as the software moving between these groups.
The tester’s job, in automating, is to shorten the feedback loop to the developer and to maintain those tests over time so that the loop remains short. This can be the most difficult part of the job, in fact, as tests break when changes are made. In the end, the only real solution is automated test maintenance, using artificial intelligence and machine learning.
These, however, are the F-1 cars of testing. While they can help anyone go faster, ML and AI-driven testing enable professionals to win races. And that’s the end goal of any software team these days: to beat the competition to implementing the latest feature.
The key to doing that is ensuring that your fast-moving development team, isn’t breaking the line of business applications in order to beat the other guy. And the only way to do that is through better testing through automation.