AI self-healing tests - locator hype vs environment flake (February 2024)

“You no longer need to write stable locators”

It is 15 February. A month ago, in the post about agents, I said explicitly what that piece would not cover: self-healing locators, because healing a selector is a separate category of tools, older than this year’s wave of agents. I promised to return to it separately, and I am doing so because since that post, in four conversations about a red E2E, I have heard the same sentence: you no longer need to write stable locators, AI will replace the selector when the page changes.

That sentence is tempting, and not because someone wrote it cleverly. It is tempting because it touches a real pain. A brittle locator is not an invented problem: one restyling in which the template gets new class names is enough to turn ten files red at once in the nightly run and consume the morning with mechanically rewriting selectors, without a single thought about the product. Anyone who has been through this wants a button that makes it happen automatically.

My thesis for February 2024 is narrow, and I will defend it throughout this post. Self-healing is an old SaaS and OSS product, resold as AI, not a new button in Playwright. Locator healing fixes one class of failure and does not touch flake from the environment, data, or a race. It also does not remove the review that in July I required for hypotheses from the log; it only moves that review to a place where it is harder to perform, because the substitution happens at runtime, not in my commit.

Here is the state of the workshop today, so a year from now it will be clear what I mean. The runner is Playwright 1.41.1 from 19 January. There is no official self-healing in this version, and I return to that in the Playwright section with the release notes in hand. I have had Copilot Chat in stable VS Code since 29 December, and it is unsuitable for healing locators for a prosaic reason: it sees the open file, not the DOM from the nightly run. The models are still GPT-4 and GPT-4 Turbo from the November DevDay. Today, 15 February, Google announced Gemini 1.5 Pro in limited preview, and I note that date only because it falls on the same day as my slot, since I did not do a single thing described below with that model.

What will not be here. There will be no sentence saying that Playwright got a healer in core, because it did not. There will be no tutorial presenting Healenium, Testim, or Mabl as February novelties, because all three products are older than the word used to sell them today. There will be no healer connected to my CI. There will be one synthetic login demo in my own repository of blog examples, thirty lines of a heuristic I wrote myself, and two red results: the heuristic fixes one and hides the other.

Three products older than the hype

The wave being discussed today as a 2024 novelty has dates. I am listing them because in conversations all three tools land in one bag labeled “new AI”, and none of them is new in that bag.

Testim was selling Smart Locators long before this year’s wave. “AI locators” appear in the announcement of the company’s acquisition by Tricentis dated 9 February 2022, two years and six days before this post. This is not, therefore, a product younger than ChatGPT. It is a product that already had a place in a larger vendor’s price list on the day ChatGPT launched.

Mabl has existed since 2017, and its auto-healing is described in the product blog, not in a February 2024 changelog. The mechanism was named exactly as it works there: the tool remembers multiple identifiers for the same element and, when one of them stops matching, adapts the step to the one that still matches.

Healenium is the same idea in open source and in a Selenium shape. It is a proxy and wrapper: it intercepts findElement, and when it receives NoSuchElement, it searches the saved tree for the most similar node from the previous healthy run and substitutes it for the locator that failed. The hub on EPAM Solutions Hub has had a 3.x line since 9 October 2020, when 3.0.1 was released, and 3.1 dates from 19 May 2021; the repository in the GitHub organization was created on 7 October 2021. I am not pinning a February 2024 patch because I do not have a URL that would confirm it today, and the dates I do have are enough to support the thesis.

The thesis is simple: none of these three products came out in 2024. What came out was hype in which a heuristic operating on a DOM tree got a new word and entered the same conversation as January’s agents.

It is worth naming this mechanism because once it is named, there is surprisingly little “AI” in it. All three tools do three things in the same order. They remember more than one way to find an element, usually during a green run: identifier, text, role, attributes, position in the tree, neighborhood. They detect a miss, meaning no match or an exception. They choose the most similar candidate from what remains on the page, according to their own similarity function. This is weighted tree comparison, not reasoning about the product, and it worked in all three tools before anyone was selling QA through a language model.

There is a reason this promise returns every few years in new packaging, and it does not involve bad faith. A test SaaS vendor does not sell assertions but suite maintenance, and “your tests repair themselves” is the best sentence that can be written on the product page of something billed for maintenance. Healenium sells the same sentence without an invoice. I am only pointing out that the sentence about self-repair says nothing about what happens to the signal.

The category difference from January is sharp, and I want to record it because the two topics blur together in conversations. An agent gets a goal and chooses its own tools; a healer gets a failing locator and chooses another one that matches. The loop from my January demo invented a data-testid that did not exist in the form. A healer makes the opposite mistake and ends up in the same place: it finds something that really exists on the page and considers it equivalent to what I was looking for. Both can produce a green result without knowing why it was red, and both deserve review like a diff, not trust like an assertion.

What a healer fixes, and what it will not touch

Healing fixes one class of failure, and it deserves credit because that class is common and boring. The id, class, or path in the tree changed, while the element is still on the page, still has the same role, and still does the same thing. During a restyling, this kind of failure takes down dozens of tests at once, and no one in their right mind would call it a product bug.

I have been fighting this class since August 2021, and it is worth noting that a healer solves it from the opposite direction than I do. The conclusion from that refactor was this: a locator is a description of how to find an element, resolved at the moment of action, not a handle saved earlier. That is why PageFactory and [CacheLookup] were removed then. A healer takes the same problem and says: leave the brittle description, and I will replace it on the fly. One direction makes the description withstand a change in the tree. The other makes the change in the tree invisible.

I have my own debt here, and I will not pretend otherwise. In my first Playwright project from May 2022, I wrote page.locator('.new-todo'), which is a CSS class locator and exactly the kind of expression a restyling breaks. I have been correcting that ever since with a role and an attribute dedicated to tests, and it is a few hours of work in one file, not a feature to buy.

Now for the things healing will not touch because they are not locators.

It does not fix a network timeout. When the login API does not return within fifteen seconds, there is nothing in the tree to substitute because the tree is fine. It does not fix shared data, and this is my favorite counterexample because I reproduced it: the flake I described in July was in a discount code with a usage counter, consumed by successive nightly runs on a persistent environment. No selector substitution touches it because the button was where it always was, only disabled by server-side validation. The rule from cleaning up xUnit and NUnit is the same here as it was in January 2022: first determine where the state lives, then choose the tools.

It does not fix a race. When the button is in the DOM but still disabled because hydration did not finish before the click, a healer will find the element with no trouble, and the click will still do nothing. It also does not fix differences on a CI agent. A font, viewport, and a slice of the environment that my laptop does not have are matters of rendering and isolation, the subject I covered in parallel tests on CI and Testcontainers on Azure Pipelines. Environment flake remains environment flake, and the only thing that moves it is an isolated dependency started for the duration of the job.

One row out of five. The September split by trigger also shows where these rows hurt me: the gate takes unit tests, contracts, and one smoke test, so this entire table, apart from that one smoke test, plays out in the nightly run, and that is where a healer would work without supervision. The pixel from November 2022 does not help me tell them apart: a screenshot says the appearance changed, a healer says the selector changed, and neither says whether the product changed.

I will show this with one synthetic login test. The repository is mine and contains blog examples, the fixture application starts alongside it with the same compose file, and the data is invented.

test('login lets a user in with the correct password', async ({ page }) => {
  await page.goto('/login')
  await page.locator('#email').fill('test@test.com')
  await page.locator('#password').fill('test')
  await page.locator('#login-submit').click()
  await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible()
})

Variant A is an identifier change. The button gets id="submit-login" instead of id="login-submit", the text remains “Log in”, and the role remains button. The test fails on the locator:

  1) [chromium] › auth/login.spec.ts:12:3 › login lets a user in with the correct password

    TimeoutError: locator.click: Timeout 15000ms exceeded.
    =========================== logs ===========================
    waiting for locator('#login-submit')
    ============================================================

I do not have a healer in Playwright, so I wrote my own heuristic to see what the report looks like after a substitution. Thirty lines, zero models, one role and one name as a fallback way to find the element:

// helpers/healing.ts - my own heuristic, not a Playwright feature
import { expect, type Page } from '@playwright/test'

type Hint = { role: 'button' | 'link'; name: string }

export async function healingClick(page: Page, selector: string, hint: Hint) {
  const primary = page.locator(selector)
  if ((await primary.count()) > 0) {
    await primary.click()
    return
  }
  const healed = page.getByRole(hint.role, { name: hint.name })
  console.log(`[heal] ${selector} -> role=${hint.role} name="${hint.name}"`)
  await healed.click()
}

export async function healingVisible(page: Page, role: 'heading', name: string) {
  const exact = page.getByRole(role, { name })
  if ((await exact.count()) > 0) {
    await expect(exact).toBeVisible()
    return
  }
  const relaxed = page.getByRole(role).first()
  console.log(`[heal] ${role}:"${name}" -> ${role}:"${await relaxed.innerText()}"`)
  await expect(relaxed).toBeVisible()
}

After replacing click with healingClick, variant A passes. The element was found by role and name, the test is green, and the entire trace of this operation is one line that I printed to standard output myself:

[heal] #login-submit -> role=button name="Log in"
  1 passed (3.8s)

And this is the only case in which the mechanism does exactly what it promises. The bill is immediately visible, however: the nightly run report does not say that the product changed the login button identifier. If the change was accidental, no one will hear about it. If it was intentional and breaks the automation contract, the contract has just been silently rewritten, at night, without a commit.

Variant B is the same red result with a completely different cause. The identifier is unchanged, but /auth in the nightly environment does not respond. The button is in the DOM, the click executes, and login does not finish, so the assertion about the dashboard fails:

    TimeoutError: expect(locator).toBeVisible() failed

    Locator: getByRole('heading', { name: 'Dashboard' })
    Expected: visible
    Timeout:  5000ms

Here healingClick has nothing to do because the button is in place. But Healenium-shaped healing does not sit on the click, only on every element lookup, so I extended the heuristic to the assertion and ran it again:

[heal] heading:"Dashboard" -> heading:"Log in"
  1 passed (4.1s)

The test that was supposed to tell me whether login works told me that the login page has a heading. Green, under five seconds, without a single red mark in the report. Variant B is environment flake, and substituting the locator here is a category error, exactly the same one the loop made in January when, instead of checking the flag in the configuration, it added an attribute to the test that did not exist.

I ran both red results twenty times in a row, once with the heuristic and once without it:

Two rows, one heuristic, two different truths, and one report that does not tell them apart. I draw a conclusion from this exercise that I did not expect: the difficult part of self-healing was never finding a fallback locator. That took me thirty lines and one evening. The difficult part is deciding whether the substitution is allowed, and that decision exists neither in my helper, nor in a SaaS price list, nor in the tree saved by Healenium.

Playwright 1.41.1 has no healer

Playwright waits for an element, retries an assertion, and records a trace. Those are three good things, and none of them is self-healing.

Auto-waiting waits for the same locator until the element is visible, enabled, and stable. Healing substitutes another locator. The first is patience, the second is substitution, and this is not a difference of degree but of kind. Assertion retries work the same way: throughout the timeout they check the same condition rather than looking for a condition that will pass.

I read the release notes for the v1.41.1 tag and found neither self-healing nor an agent that heals locators. I will not pin a feature that is absent from the release notes, because that is exactly how myths are later made. If someone sells “AI locators for Playwright” as a plugin or service, it is a separate product, with a separate review and a separate place in the contract, not a feature of the runner I have in package.json.

What I actually do when a locator fails has not changed since I first had a trace. I open the trace and read whether the element was in the tree, whether it was invisible, whether it was covered, and what the network was doing at the time. In variant A, the DOM snapshot shows a button with a new identifier, and that is a two-second verdict. In variant B, the same snapshot shows the button in its usual place, while the requests tab shows a hanging /auth, and that is a verdict about the environment, not the test. Two clicks, two different answers, and the trace calls neither of them “healed” because it has no such concept.

There is also a case in which the similarity heuristic is genuinely useful, and I do not want to pretend otherwise. A mass identifier change after a restyling, thirty failing tests, all with the same cause. Then I want a list of proposals in a pull request, run by me on demand, with a diff for every file and my approval on the whole thing. That is a codemod, not a healer. The difference is who decides and when: once, consciously, in the repository, not thirty times, at night, at runtime.

I ask three questions about every red test, and none of them is “can this be healed”. Does this test check anything at all, or does it merely wait for an element. Is the change in the product or in the environment. Can I reproduce the failure before proposing a fix. This is the same set I left in January beside the agent loop and in July beside hypotheses from the log. Healing answers a fourth question that I did not ask: is there another selector with which this step will pass. During a refactor, that question is useful. As an automatic in the nightly run, it is harmful.

When a healer enters the team anyway

I will not ban this tool in someone else’s team, and I have no intention of doing so, because with a mass identifier change I understand where the demand comes from. I can say under which conditions it will not erase the signal, and I have four.

First: the substitution is an event in the report, not in a log that one person reads. The old locator, the new locator, a snapshot of the element the new one matched, and the name of the test that passed because of it. I review it like a diff: first I reproduce, then I agree or cross it out. In my demo, one line on standard output played this role, and losing it among forty green tests was enough to make the entire story cease to exist.

Second: the substitution expires. A heal that survives five nightly runs is no longer a heal but a silent rewrite of the test. After it expires, I want a red result, exactly what the healer was supposed to remove, because only red forces a decision on whether the new locator should enter the repository.

Third: healing does not enter the gate and does not touch assertions. After the September split, the things in the gate are fast and deterministic, and a mechanism that, on the same input, sometimes passes with a locator from the file and sometimes with a locator it chose itself is not deterministic. Assertions are excluded because of variant B: the moment the target of an assertion is subject to “healing” is the moment the test stops asserting anything.

Fourth: it has an owner. In the 2023 recap, I recorded four things that tell me a tool has entered the team rather than the hype: a named task, a place in the process, a row in the pipeline or a conscious reason why it is absent, and a person accountable for the outcome. A healer has the first if we limit it to “suggest another locator during a refactor”. It does not have the fourth if it rewrites the suite by itself at night.

The contrast with July is mandatory here because these two topics are confused most often. In flake triage, the model received a fragment of a log and returned three hypotheses. Of thirty-three hypotheses, I confirmed nine by reproduction and rejected sixteen, and the one ranked as the most likely in my main example turned out to be false. One thing is key: none of them changed a single line of the test without my commit. Self-healing works in the opposite direction. It changes the locator at runtime, while the log, if one is created at all, describes success, not the cause. This is not triage. It is an automatic fix without an owner, and with something like that my July notebook would be impossible because I would have nothing to reproduce.

Summary

Self-healing is this month’s topic because the word “AI” stuck to products I have known for years. In February 2024, I am not buying it as a substitute for isolation and review.

Three sentences I want to be able to read a year from now. Testim advertised Smart Locators in the acquisition announcement as early as 9 February 2022, Mabl has existed since 2017, and Healenium’s 3.x line has its first release known to me dated 9 October 2020, so this year’s hype concerns the mechanism, not its launch. Playwright 1.41.1 has no official self-healing, and auto-waiting is not self-healing because it waits for the same locator and does not choose another one. Thirty lines of my own heuristic fixed the id change in twenty runs out of twenty and, in the same twenty, hid the /auth timeout behind a false green, so the cost was not in the quality of the mechanism but in the report losing the sentence about whether the product changed.

What does not follow from this. It does not follow that Smart Locators have no value, because after a restyling that rewrites half the template, a list of proposals in a pull request saves real hours. It does not follow that I will never put a “find the same button by role and name” heuristic into a helper, because I have just written it and am leaving it in the examples repository as a tool to run manually. Nor does it follow that this will not change if a healer someday enters the runner’s core; it is not there today, so I have nothing to write about except the date.

The condition by which I will know the topic has matured is this: a locator substitution is acceptable when it leaves an auditable diff, expires after a few runs, never touches the target of an assertion, and does not take away from a person the question of whether the red test caught the product or the environment. Until then, my suite has no autonomous healer. It has isolation, a trace, and me.