Visual regression in 2022 - Playwright, Backstop, Percy, Applitools

The promise from August

In August, when describing the pipeline with containers and Playwright, I listed the things I was deliberately not adding to the e2e job. Image comparison was one of them. I wrote then that screenshots in CI mean a discussion about tolerance thresholds, rendering differences between machines, and who approves new baselines, so adding them to a ready pipeline without those decisions would give me red runs with no information. I stand by that statement. Today I only want to finally have that discussion.

I have described visual regression on the blog exactly once, in March 2020, with a free plugin for Cypress. The question then was narrow: can I compare screenshots in a tool I already have without paying a single zloty for it? The answer was yes, the plugin caught a removed period in the sentence “No articles are here… yet.”, and that post ended with one open topic: CI.

Two and a half years later, the question is different because the landscape has changed. Playwright has had a screenshot assertion in the runner core since May. BackstopJS, which is older than my entire series about Cypress, has been able to run on the Playwright engine since last November. Percy and Applitools have SDKs for Playwright, one since October last year, the other since August 2020. So instead of one plugin, I have four paths plus a fifth, historical one. This post is about choosing between them.

I will set the boundary of this article right away: this is not a tutorial. I am not rewriting the 2020 post for Playwright, and I am not building a complete visual suite for Conduit. The listings below are short and intended to show the shape of the API, not replace the documentation. I am interested in the trade-off matrix, because that is what determines whether image comparison is still useful to anyone after three months or has been disabled with a flag.

Three models, not five tools

Before listing the packages, I will describe the axis along which they lie. The tools differ in their APIs, but the real choice is between three models, and that choice is what costs.

Baseline in the repository. The PNG file sits next to the test, git versions it, and approving a new look is a commit. This is how Playwright core works, how Backstop works, how my 2020 plugin worked, and how the open source lost-pixel works. The license costs zero, but I pay in three other ways: the repository grows with binaries, reviewing a visual change looks like reviewing a binary file, and the result depends on the machine that rendered it.

Baseline in a service. Screenshots go to a service, the diff and approval happen in a web UI, and the baseline is attached to the branch on the provider’s side. This is how Percy works. I pay with money and with the fact that images of my application leave for someone else’s server, and in return I get something git does not provide: a non-technical person can click “approve”.

A comparison that is not pixel by pixel. Applitools sells Eyes specifically as comparison algorithms, with modes that can ignore shifts in content or changes in data while catching a layout change. This is an algorithm embedded in the product and run on their side, not a tool that decides on its own whether a change is desirable. The distinction matters because the marketing term “AI” in this category means a specific, deterministic way of calculating the difference, and that is how I treat it here.

The criterion that weighs the most in this division is not accuracy, but approval. Each of these tools will eventually light up red because of a change that is correct. The question is how much it costs to move such a change forward. In the git model, that means overwriting the baseline and committing it; in the service model, it means clicking in a browser. When the cost of approval is high, people start ignoring the result, and we return to the problem I described in September when discussing test strategy: a gate that nobody reads is not a gate.

Playwright 1.27: a core assertion

I am starting with the cheapest option because I already have it installed. My @playwright/test is at 1.27.1 from 12 October, and this is the only version on which I checked anything here.

The toHaveScreenshot() assertion has been in the tool since 1.22.0 from 12 May, which is exactly the release on which I set up an empty project and about which I wrote one sentence at the time saying that visual regression was a separate topic. So this is not a November novelty, but a feature that is six months old and has had time to settle down.

import { test, expect } from '@playwright/test'

test('login screen matches the baseline', async ({ page }) => {
  await page.goto('/login')

  await expect(page).toHaveScreenshot('login.png', {
    maxDiffPixelRatio: 0.01,
  })
})

Three things in this listing make a difference compared with what I had in 2020. First, the assertion is web-first: the runner takes a screenshot, waits, takes another, and compares them with each other until the page stops moving, only then comparing the result with the baseline. Second, animations and the cursor are disabled by default, so I do not have to kill CSS transitions with my own style. Third, I provide the threshold explicitly, in maxDiffPixelRatio or maxDiffPixels, and this is a number worth deciding once for the project instead of adjusting it after every red run.

Baselines land next to the test file, in a directory with the -snapshots suffix, and the filename includes the project and system, for example login-chromium-linux.png. Updating them takes one flag:

$ npx playwright test --update-snapshots

The consequence of this naming is more important than it looks: the baseline is platform-specific. A screenshot generated on my Linux machine will not match a screenshot from a colleague’s macOS because font antialiasing differs, and sometimes the fonts themselves differ. The only sensible solution is to generate and verify baselines in the same container image in which CI runs, which is exactly what I configured with Testcontainers on the agent and in the August pipeline. Anyone who does not do this will get a tool that turns red with every machine change.

The older API is not disappearing. page.screenshot() still serves for artifacts, and expect(buffer).toMatchSnapshot() still exists and is more general because it compares any buffer, not only an image of the page. For visual regression, however, I choose toHaveScreenshot(), precisely because it waits for stabilization, which I have to manage myself in the other variant.

BackstopJS 6.1.4

Backstop has been on npm since 2015, and that is its first advantage: it is not an experiment. The version from 4 November is 6.1.4, and since 6.0.1 from 18 November last year the tool has been able to use Playwright as its engine instead of Puppeteer.

The difference from the previous section is fundamental and is not about the quality of the diff. Backstop is a separate runner with its own configuration file in which I describe a list of addresses and a grid of resolutions. I do not write tests, only scenarios:

{
  "id": "conduit",
  "engine": "playwright",
  "viewports": [
    { "label": "phone", "width": 375, "height": 667 },
    { "label": "desktop", "width": 1366, "height": 768 }
  ],
  "scenarios": [
    {
      "label": "login",
      "url": "http://localhost:4100/login",
      "misMatchThreshold": 0.1
    },
    {
      "label": "article list",
      "url": "http://localhost:4100/",
      "misMatchThreshold": 0.1
    }
  ],
  "paths": {
    "bitmaps_reference": "backstop_data/bitmaps_reference"
  }
}

$ npx backstop reference
$ npx backstop test
$ npx backstop approve

I pay for this by having a second runner in the project. The playwright.config.ts configuration does not apply in backstop.json, baseURL has to be provided again, and logging in or preparing data requires separate onBefore and onReady scripts, so all the work I have already done in page objects and login helpers cannot be used there without rewriting it.

That is why I see one specific place for Backstop: when I want a grid of many addresses at several screen widths and I do not want to write tests for it. A marketing site, a style library, a collection of landing pages. For an application where I have to log in and click through three steps, I prefer to stay in the runner where I have already described those three steps. At the same time, the HTML report with three panels - baseline, current state, and diff - is the clearest of the entire open source group.

Percy

@percy/playwright came out in 1.0.0 on 20 October last year, and since 1.0.4 from 24 May it has not had a newer stable release, so this is the version I pin today. The code in the test is shorter than everything above:

import { test } from '@playwright/test'
import percySnapshot from '@percy/playwright'

test('login screen', async ({ page }) => {
  await page.goto('/login')
  await percySnapshot(page, 'Login')
})

$ PERCY_TOKEN=... npx percy exec -- npx playwright test

There is no threshold in the test, no filename, and no assertion. This is not an oversight in the listing, but the whole model: percySnapshot collects the state of the page and sends it to the service, while the comparison and the decision happen outside the run. A run containing only snapshots cannot be red for a visual reason, because at that point nobody has compared anything yet. The result returns as a status on the pull request on GitHub, and that is where someone clicks approval.

There are two advantages, and both are organizational, not technical. The baseline is not a file in my repository, so I do not have to decide who commits the binaries and what to do about a conflict in a PNG. Approval can be clicked by someone who does not have the repository on their disk, meaning a designer or product owner, and this is the only way I know for a visual change to be approved by someone who decides how it should look.

There are also two disadvantages. First, images of my application leave for an external service, and that is a discussion to have before a pilot, not after. Second, this is a service billed by snapshots, so the number of screenshots times the number of screen widths times the number of runs is a budget, not a configuration detail. I do not give any prices here because pricing has a life of its own while the post stays on the internet for years. Instead, I give a rule: count your snapshots before enabling this on a nightly schedule, because the matrix multiplier can be surprising.

Applitools Eyes

@applitools/eyes-playwright has existed since 1.0.0 from 5 August 2020, and the current version today is 1.12.7 from 11 October. The API is more elaborate than Percy’s because this is not a single call, but a session lifecycle:

import { test } from '@playwright/test'
import { Eyes, Target, VisualGridRunner } from '@applitools/eyes-playwright'

const runner = new VisualGridRunner()

test('login screen', async ({ page }) => {
  const eyes = new Eyes(runner)
  await eyes.open(page, 'Conduit', 'login screen')

  await page.goto('/login')
  await eyes.check('form', Target.window().fully())

  await eyes.close()
})

Two things distinguish this from the rest. The first is Ultrafast Grid, represented by VisualGridRunner in the listing: the local browser captures the page once, while rendering in multiple browsers and resolutions happens on the service side. Instead of running my own matrix from the post about cross-browser testing and multiplying the run time by the number of combinations, I send one data set and receive results for many configurations. With ten combinations, this is a real difference in wall-clock time.

The second is match modes. The default is not a simple pixel-by-pixel comparison, while Layout mode deliberately ignores content and looks at the arrangement of elements. This is the answer to a problem that a tolerance-threshold approach handles badly: a page with a dynamic date, a counter, or a random record order will be red every time, while raising the threshold enough to let it through will also let a real defect through. I have not spent months working with this mechanism, and I will not pretend that I have. I note it as something to check on an application with a large amount of variable data, because that is the only place where it has a chance to beat a simple diff.

The cost here is of the same nature as with Percy, except that the licensing model is more corporate and usually ends in a sales conversation rather than entering a card number. Again, no prices.

The Cypress path, honestly

That leaves the plugin from the 2020 post, namely cypress-image-snapshot. If someone has a test suite in Cypress today and wants to compare images, this is the shortest path because cy.matchImageSnapshot() can be inserted into an existing test with one line, and everything else remains unchanged.

However, I have to say something here that could not have been in that post. The last release of this plugin, 4.0.1, is from January 2021, and Cypress has gone through two major releases since then. Version ten from 1 June changed the configuration structure and the plugin model, so the cypress/plugins/index.js file into which I pasted addMatchImageSnapshotPlugin in 2020 is simply no longer an entry point in the new structure. Version eleven, which I have today at 11.0.1 from 10 November, continues in the same direction.

I have not checked whether 4.0.1 runs on Cypress 11 without a fork, and I will not write that this is a drop-in installation because I do not know. The honest version is this: the 2020 recipe applies to Cypress from that era, and it works there. On version ten and above, the first step is to check whether the plugin is still maintained and whether someone has migrated it to setupNodeEvents, not to copy my old listing. If the answer is negative, that is not a reason to rewrite the whole suite. It is a reason to write new visual tests where this feature is in the tool’s core, namely in Playwright, exactly in the spirit of the April decision: new areas go to the new tool, while old green tests stay where they are.

lost-pixel, or a note for the future

I mention it so it does not look like I overlooked it. lost-pixel has been on npm since 4 June this year, and the current version is 2.23.0 from 6 October. It takes screenshots from Storybook or Ladle, compares them, and has a ready-made GitHub action, so it looks neat for a project with a component library.

And I will leave it at that. The package has been on npm for five months, the hosted platform has only been announced, and today I am comparing tools on which I want to base something that should last a year. I am putting it on the list to look at next year; I am not placing it alongside Percy as an alternative, and I am not recommending that anyone replace a working Backstop setup with it.

While I am here, there is one thing from the same shelf that I am also not considering: component testing in Playwright. The @playwright/experimental-ct-* packages have been in the tool since May and still have a word in their name that explains everything. I am not building component visual regression on this in November 2022.

Where it lives in the pipeline

The choice of tool matters less than where I run it, so I am adding this separately.

Image comparison is not a required check on a pull request. For me, it goes into the third bucket from the September breakdown, namely the nightly schedule, and not because it is slow, but because it inherently depends on the environment. A visual test that blocks a merge will become a weather phenomenon after two weeks, not a gate.

Screenshots and diffs have to leave the agent as artifacts, otherwise a red result is useless. The mechanics are the same as what I configured with Cypress on Azure DevOps; only the list of paths to package changes.

I keep all visual runs in one container image and on one engine. The baseline is tied to the platform, and parallelism across several machines only makes sense if all slices render in the same way. I functionally test three browsers, but visually only one for now, because three times as many baselines means three times as many approvals.

And one thing that is easy to forget after a month spent on the API layer. The in-process integration tests that I described in October are fast and deterministic, and they will say absolutely nothing about someone changing a button class and making the form disappear under the footer. They answer different questions, and one does not replace the other.

How I choose in November 2022

This leads to four decisions that I am making today.

A small or medium-sized suite in TypeScript, one application, a development team: I stay with Playwright core. Zero new dependencies, the baseline next to the test, the threshold in one place, and the same command as for the rest of the tests.

A grid of many addresses at several widths, without login and without scenarios: Backstop. The configuration is declarative, the report is clear, and the fact that it is a separate runner stops hurting when I have nothing to attach it to anyway.

An appearance decided by someone outside the repository, or a page with a large amount of variable data: a service. Percy when the most important thing is a simple approval flow; Eyes when match modes and rendering the matrix on their side matter most. In both cases, I start by counting snapshots and obtaining approval to send screenshots outside.

A working suite in Cypress: I do not touch it. The 2020 plugin stays where it is as long as it is on the Cypress version for which it was created. I add new visual tests in Playwright and do not turn this into a migration.

Summary

Three things I take away from this comparison.

The cheapest tool in this category is not the one that is free, but the one in which approving a correct change costs the least. Everything else is decided second.

The baseline is a project artifact, not a side effect of a test. Kept in git, it is tied to the platform and requires discipline about the machine that produced it. Kept in a service, it requires a budget and approval to send images of the application outside. There is no third option, and choosing one of the two is precisely the decision I had been postponing since August.

Visual regression does not answer the question of what is worth testing. It answers only whether it looks the way it did yesterday. A year ago, in my year-end summary, I noted that isolation comes first and everything else follows, and the same order applies here: a suite that is functionally flaky will be flaky twice over after images are added. What deserves to be tested through the interface at all remains a separate discussion that none of these five packages will have for me.

What I do not know after this week of reading: whether the match modes in Eyes can really handle an application with a large amount of variable data, and whether the maxDiffPixelRatio threshold I entered off the top of my head today will survive the first month in the nightly run. I will check both values on my own suite and write about them then.