5 Reasons Why Test Automation Used to Be So Expensive

As mentioned in the previous post the transition between backend and front-end has been a major transformation. During the mainframe…

Test Automation Costs
Testim
By Testim,

As mentioned in the previous post the transition between backend and front-end has been a major transformation.

During the mainframe era, most code resided on the server, moving to the client when personal computers came into play. It shifted back to the servers with the web, and in the last five years, we’re seeing code shifting back to the client side as Single Page Applications (SPA) become more popular (and I assume most of you already heard of Angular or React or Ember or Vue and list goes bigger every day).

For back-end testing, most find API testing sufficient as it tests with high fidelity most of the server code. Unit Tests weren’t popular until recent years

As for front-end testing, little code resided on the client side, and API (backend) testing covered most of the app’s code. Many companies historically relied on manual testing. Automation was expensive due to 5 main reasons:

  1. Highly trained people – they are developers in any way: They write complicated code; They spring VMs and browsers on and off and deal with large scale when it comes to performance testing.
  2. Finding great developers is hard. Finding great developers who want to write test automation for a living has the same chances of Shaquille O’Neal not missing a single free throw in ten consecutive shots. 40% of a test automation person time is spent on authoring tests (1).
  3. High maintenance – In the agile world, UI changes are frequent, This implies that changing the UI requires changes in the UI-tests. 30% of a test automation person time is spent on maintaining tests (1).
  4. Long ramp-up – it usually takes weeks or months until you have a dozen of tests.
  5. Few environments. The OS and browsers were not as dispersed as today. Windows ruled the Desktop world, and IE had 95% market share.

Record/Playback

Many platforms tried to alleviate slow authoring and the required dev skills by recording the user’s behavior and creating a test, with the ability to play it back again and again. Unfortunately, those tests required massive maintenance, making the total ROI even lower than with dedicated developers.

Visual Validation

Visual Validation – an image is worth 1000 validations

The complexity of the UI, as opposed to API testing, crosses more than functional testing (the shown data) but also includes visual validation, as visual bugs are not merely cosmetic bugs. They impact usability and user experience. As UI validations were usually associated with E2E testing, most attempts were to take screenshots of the entire page as a validation (as opposed to a small component). Those attempts suffered from the next set of challenges:

  1. Stability – Computers’ display adapter are non-deterministic by nature, and rendering the same page twice might result in different images. The main diffs stem mostly from anti-aliasing and sub-pixels shifting. The human eyes won’t notice, but a simple pixel comparison might result in a  5% difference.
  2. Maintenance – Changing a single component which is located across pages (e.g. logo) can imply that hundreds of baseline images are not incorrect and has to be reviewed.
  3. Storage – Maintaining huge volumes of large screenshots in a non-lossy format were something heavy to any version control repository, specifically the well adopted Git.

Visual validation became scarce until recently.
We’ll mention some solutions in the next blog post, stay tuned!

[1] Source: http://www.softwaretestingnews.co.uk/res/14072016/TEST_Benchmark-Report-2016.pdf