A second channel, not a second Percy
It is 15 May. In November 2022 I compared what a look gate was then: Playwright toHaveScreenshot, Backstop, Percy, Applitools Eyes. That post still describes my nightly. Today I add a second channel and I want nobody to mix it with the first. I paste a PNG into a vision model and I get sentences: “the button drifted right”, “the date banner covers the CTA”. That is a description. That is not exit 1.
My thesis for May 2024 is narrow. An LLM with vision can describe a screenshot. It does not replace a pixel or layout gate. In Playwright 1.44.0 from 6 May, toHaveScreenshot still fails the build. The model’s comment stays a review that I read.
Workshop state. The runner is 1.44.0. I keep snapshots in git and I run --update-snapshots on purpose. Since DevDay on 6 November 2023 I have gpt-4-vision-preview; since the 9 April notices the gpt-4-turbo alias with vision has been circulating, and I do not rest a sole GA date on the press. Claude 3 has been able to look at an image since 4 March. On 13 May, two days ago, OpenAI announced GPT-4o as better at vision. I am not moving nightly onto it and I am not making it the core of this post.
What will not be here. There will be no “PNG into ChatGPT = visual regression in CI”. There will be no 2022 matrix again. There will be no official “Playwright LLM screenshot assert”, because 1.44.0 does not have one. There will be a synthetic baseline and an actual with a shifted CTA.
What I already have as a gate
My look gate since May 2022 and since 1.22 is toHaveScreenshot. In 1.44 the contract did not change: compare against a baseline in the repository, fail when pixels drift past the threshold, a diff in the report. That is the same category as Percy and Eyes from November 2022, except Eyes sells “Visual AI” as its own engine, not as a chat I paste a PNG into.
A fail is a build verdict. A model opinion is not. I wrote that line in September 2022 for the gate and the nightly, and I will not blur it because a model describes a hero nicely.
What I do with a model
I take two PNGs from a synthetic marketing page in the examples repository: Friday’s baseline and Monday’s actual. The question I paste is deliberately narrow:
These are baseline and actual of the same hero.
Is this a layout regression, or only a data-test banner?
Do not propose a pixel threshold. Write three sentences.I treat the output like a hypothesis from July 2023: a list of things to check, not a merge. Sometimes I paste only the actual when I know what should be there. I do not pin this in Actions as a required check. I do not upload production screenshots with PII.
GPT-4o is two days old and looks more confident on small text in demos. On 15 May I do not have a week of measurements from it, so I leave it at one sentence. My pair for describing a PNG is still gpt-4-turbo or Claude 3, plus toHaveScreenshot beside them.
Where the model lies
Demo: I shifted the hero CTA by one grid column. toHaveScreenshot fails. That is a correct red.
The model wrote three sentences. Two were useful: the date banner on the actual is new, the CTA is no longer in the same column as the baseline. The third was wrong: “it looks OK, that is only copy”. On a 2px font shift from CI it can also say “OK” while the grid no longer holds. It hallucinates button text that is not in the PNG, exactly as in April Chat invented a data-testid.
That is why an LLM is not a required check. A pixel gate does not ask whether it “looks OK”. It asks whether the pixels match. Eyes and Percy from 2022 are not chat either. They are tools with a baseline and a threshold.
When it is worth it at all
A one-off review of “what changed on the marketing hero” when the baseline is old and the pixel diff is large, and I want sentences, not noise. Not a nightly of two hundred viewports. Nightly stays on toHaveScreenshot or on Percy / Eyes I already have in the process. The pyramid does not get a new “LLM screenshot” layer.
I do not mix in component testing from August 2023: CT is mounting a component, not visual E2E of a hero.
Summary
Vision speeds up reading a screenshot. The gate in May 2024 remains the 2022 tool plus Playwright 1.44.
Three sentences for a year from now. toHaveScreenshot on 1.44.0 still fails the build on a shifted CTA. The model describes the same PNG in three sentences, one of which lies that the layout is fine. The required check stays the pixel, not chat.
What does not follow. It does not follow that GPT-4o will not enter this channel in a month. It does not follow that Eyes is chat. It does not follow that I will delete the baseline.
The maturity condition: two descriptions in a row on the same diff do not contradict the pixel gate and do not invent button text. Until then I describe PNGs with a model and I merge only when a snapshot or a human says that is how it should be.