A pretty button, an empty tree
It is 15 August. In November 2022 I compared look gates: toHaveScreenshot, Percy, Eyes, Backstop. In May I added a model describing a PNG, and the gate stayed the pixel. Today I open the same Playwright suite and see a different hole. The icon button on a synthetic login is aligned in the grid. The snapshot is green. In the accessibility tree there is no name. A screen reader will say “button”.
My thesis for August 2024 is narrow. An a11y automat in Playwright is two cheap signals on the same page load: the axe-core rule catalog, and a hard assertion on one control’s name or role. That does not replace a screen reader or a WCAG audit. That is not the visual gate from that November. Pixels and the tree catch different regressions.
Workshop state. The runner is Playwright 1.46.0 from 5 August. @axe-core/playwright is pinned at 4.9.1 from 20 May. I have had toHaveAccessibleName, toHaveAccessibleDescription and toHaveRole since 1.44.0 on 6 May; in April Chat proposed toHaveAccessibleName on pin 1.43.1, which did not have it. WCAG 2.2 has been a Recommendation since 5 October 2023 and stays a map, not a certificate I issue after a green job.
What will not be here. There will be no auditor course and no list of eighty success criteria. There will be no toMatchAriaSnapshot, because 1.46 does not have that assertion. There will be no pin of 1.46.1 or @axe-core/playwright 4.10.0. There will be no sentence that an LLM fixes a11y. There will be a synthetic login form: an icon with no name, an input with no <label>, AxeBuilder, and two assertions.
Two signals, not one scanner
The first signal is the catalog. On page load I run new AxeBuilder({ page }).withTags(['wcag2a', 'wcag2aa', 'wcag21aa']).analyze(). I get known patterns: missing label, contrast axe can compute, a duplicate id. I fail on a violation, not on 400 untagged rules.
The second signal is a pin on one control. expect(page.getByRole('button')).toHaveAccessibleName('Log in') does not ask the catalog whether it “looks OK”. It asks whether this specific icon has the name I expect. toHaveRole does the same for a password field someone wrapped in a div with a role.
A control can pass axe and still lose its label in a copy refactor. Axe looks at a pattern. The assertion looks at a contract. That is why both layers live in one smoke spec, not in two religions.
The demo I started this week with is synthetic and deliberately broken. The login form has an SVG button with no aria-label and a password input with no <label>. AxeBuilder returns button-name and label. toHaveAccessibleName('Log in') fails because the name is empty. Only after an aria-label on the icon and a <label for> on the password do both signals go quiet. I am not pasting a production form screenshot and I am not quoting a client audit.
Where it sits in CI
A11y smoke sits where UI smoke sits: on the PR or on main, on the login path, not on the whole app. A wider scan with a broader tag set goes on nightly. I wrote that boundary in September 2022 and I will not blur it because axe paints a nice report.
Fail on HIGH and on selected wcag2a / wcag2aa / wcag21aa tags. Not on every best-practice on every push. A job that goes red on a rule we will not fix this sprint teaches the team to ignore red.
This is not a new headcount. This is the same runner where I have had E2E since May 2022. I am not mixing in the screenshot plugin from Cypress: different axis, different engine, the same category error if I fold visual into a11y.
What the automat does not see
Focus order in the user’s head. The sense of a label that exists and lies. Contrast in a theme axe did not get as CSS. A reader that speaks a different order than the DOM. That stays with a human and with one keyboard pass.
Three false certainties from this week, all to strike out.
Strike one: “WCAG AA passed”. Green axe on three tags is not Recommendation 2.2 and not a legal finding. It is a rule catalog on one URL. The process does not end at a scanner.
Strike two: visual will catch a missing name. toHaveScreenshot from November 2022 sees the icon in the grid. It does not see that the tree has no Log in. In May a model will describe the PNG. It still will not sign aria-label.
Strike three: “I will add an ARIA snapshot and it will be a pixel gate”. On 1.46.0 I do not have toMatchAriaSnapshot. If I pasted that assertion from a future doc, I would get TypeScript or a runtime error before I got a verdict on the name. I stay with AxeBuilder and toHaveAccessibleName.
I have had GPT-4o mini since 18 July. It is not the subject of this post. I do not paste it HTML with “fix a11y” and I do not treat the answer as a merge. Cursor is not the daily driver in this workshop.
Contrast with visual
Visual catches layout. A11y catches name, role, label. I do not fold them into one “UI quality” job. 2022 gave me Playwright and visual. August 2024 adds the tree. It does not delete the pixel.
When the icon drifts a column, toHaveScreenshot fails. When it stays put and loses aria-label, a11y fails. Both reds are correct. One job that is supposed to “fix UI” hides which of them is real.
Summary
In August 2024 a cheap a11y automat is AxeBuilder 4.9 on Playwright 1.46 plus a name assertion. Green axe does not close an accessibility ticket.
Three sentences for a year. Login smoke fails when the icon has no name, even when the snapshot is green. toHaveAccessibleName pins one control’s contract, which the catalog can miss on a copy change. The reader and the human stay, because the scanner does not know what the label means.
What does not follow. It does not follow that I hold a WCAG 2.2 certificate. It does not follow that I delete visual. It does not follow that 1.46.1 already sits here, because it does not.
Maturity condition: two runs in a row on the same form agree with a keyboard pass and do not shout “AA passed” just because violations is empty.