AI in flake triage - hypotheses, not a verdict (July 2023)

A red overnight run and a chat tab

It is 15 July. In the morning, I look at the overnight run report, see one red test, and know in advance that I probably will not see it fail again today. This is the class of result where someone clicks “re-run”, gets green, and goes back to their work, while three weeks later no one reads the morning report anymore.

The flake did not disappear from my pipeline because I have had a chat tab open for six months. What did disappear was the excuse that I have no one to talk to about a log at half past seven in the morning. And that is exactly the question I want to ask in this post: does the model shorten the first, most mechanical stage of triage, where I have to decide which of four buckets an incident belongs in? The test is wrong. The data is wrong. The infrastructure had a bad day. The product is broken.

This first division determines everything that happens next because it decides who gets the ticket and how much time I spend on it. It is also the stage where it is easiest to make the most expensive kind of mistake: label a product bug as “the environment” and close the matter by restarting the job.

From 6 to 14 July, I kept a notebook. I ran every red result that I did not understand within the first thirty seconds through the model, wrote down the hypotheses it gave me, and recorded what happened to them after an attempt to reproduce the failure. The result was 11 runs and 33 hypotheses, three per run. This is one person’s notebook from nine days, not a study, and that is how I will write about it.

I am establishing the state of the tools this week right away because six months from now no one will remember what was behind a gate. I have had GPT-4 in the ChatGPT Plus model switcher since March, and that is where I do all the work described in this post. Since 6 July, gpt-4 has been available in the API without a waitlist for paying customers, which means that for the first time I could script this; I did not, and I will explain why in the section about my working practices. I have had Copilot Chat since May, when I came off the waitlist and opened it in VS Code Insiders, and it is still a technical preview: a separate editor installation plus the Nightly extension. I do not use it for flake triage because chat in the editor sees the open file, not the overnight run log; all the work in this post happens in Plus. As of this post, the public beta for organizations on Copilot Business has not been released, and no one is even talking about GA. On 11 July, Anthropic announced Claude 2, and I am leaving it at one sentence because I did not run a single one of my own logs through it. My runner is current: Playwright 1.36.1 from 14 July, which was yesterday.

What will not be here: a tool that repairs a selector by itself, a model connected to CI, or anything that creates or closes an issue on its own. The model in this post does exactly one thing: it produces a list of things to check. I do the checking.

What I paste into the chat

I paste four things, and I arrived at this list through trial and error because my first conversations consisted of pasting the entire run log and asking, “what is wrong?”

The first is the full test name together with the file and line number. Without that, the model has no idea whether it is looking at a unit, integration, or browser test, and it guesses from the messages, usually incorrectly.

The second is an excerpt from the log, not the entire log. I include thirty to sixty lines around the failure, with Playwright’s logs section, the code excerpt reported by the runner itself, and the attachment list. The run header listing forty green tests contributes nothing except tokens.

The third is the test’s place in the pipeline, and this is the information people forget most often. A failure at the pull request gate and a failure in an overnight run are two different worlds for me because they have different dependencies and different environments. I described the division into buckets by trigger in September, in my CI/CD test strategy, and that is the context here: the gate runs unit tests, contracts, and one smoke test, while the overnight job runs the full E2E suite. The model cannot know that, so I tell it explicitly which bucket I am in.

The fourth is one sentence about what was in the commit. It usually says, “the commit did not touch either this test or this feature,” because if it had, I would not be doing triage; I would be reading the diff.

Then there is redaction, and it is not a formality because a CI log is full of things that must not leave the organization. I remove tokens and Authorization headers, internal service addresses and agent names, artifact addresses with signatures in the query string, real email addresses of test users, order IDs, and anything that might reveal whose project it is. The rule is the same one I wrote down in March, when using GPT-4 to draft test cases: pasting someone else’s artifact into an external service is a decision for the owner of that artifact, not mine. When redaction leaves me with a log that says nothing, that means I do not triage that failure in the chat.

I also have no place where I can set this context once and keep it. I therefore store three paragraphs in docs/triage-prompt.md in the repository and paste them at the beginning of every conversation: my stack, my pinned versions, and a ban on proposing fixes in the first response. The prompt structure is the same one I laid out in February, except the input is a log rather than a specification.

One note about the model. gpt-3.5-turbo is not enough for this task, and I checked twice. With sixty lines of logs, it starts commenting on the first third and loses what is at the end, which is precisely the attachments section and the run summary. GPT-4 in Plus keeps track of the entire pasted block, and this is the same difference I described in March with a long specification. The Plus message limit does not hurt during triage because this is not work that involves twenty questions in a row.

Three hypotheses instead of one verdict

I will show one complete run because otherwise everything that follows would be a story about impressions.

The log below comes from my own repository of examples for this blog, not from any client project. I replaced the test name, selectors, and discount code with those from the fictional store I used in March so that I would not mix the two worlds. Its shape, messages, and timings are exactly what the runner printed.

Running 40 tests using 4 workers

  1) [chromium] › cart/coupon.spec.ts:41:3 › koszyk › nalicza rabat po zastosowaniu kodu

    TimeoutError: locator.click: Timeout 15000ms exceeded.
    =========================== logs ===========================
    waiting for getByTestId('apply-coupon')
      locator resolved to <button disabled data-testid="apply-coupon">Zastosuj</button>
    attempting click action
      waiting for element to be visible, enabled and stable
        element is not enabled
      retrying click action, attempt #2
      waiting for element to be visible, enabled and stable
        element is not enabled
    ============================================================

      39 |     await cart.open()
      40 |     await cart.couponInput.fill('SUMMER10')
    > 41 |     await cart.applyCoupon.click()
         |                            ^
      42 |     await expect(cart.total).toHaveText('90,00 zł')

    attachment #1: trace (application/zip) ----------------------
    test-results/cart-coupon-nalicza-rabat-chromium/trace.zip
    ------------------------------------------------------------

  1 failed
    [chromium] › cart/coupon.spec.ts:41:3 › koszyk › nalicza rabat po zastosowaniu kodu
  39 passed (2.4m)

The prompt looked like this:

I am a QA engineer. Below is the log from one red test using Playwright 1.36.1,
TypeScript, 4 workers, an overnight run rather than a pull request gate. The
commit in this run touched neither this test nor the basket.

Do not give me one cause. Give me 3 hypotheses ordered from most to least
likely. For each, add: how I can reproduce it locally, what I will see if it
is true, and what I will see if it is false.
Do not propose fixes in this response. Do not assume code that is not in the log.

I received three hypotheses in this order:

1. An environment race on the agent. Four workers talk to one application instance on one port, so one worker sees another worker’s basket, and the button is disabled because that basket already contains the discount code.
2. A shared data fixture. SUMMER10 is a resource shared by the entire suite rather than data belonging to this test. Someone consumed it first, and server-side validation keeps the button disabled.
3. retries in the configuration have masked this for weeks. The same problem also occurs at the gate, but the second attempt passes, so no one sees it.

This is a good list. It is also a list that must not be read as a verdict, as each of its three points demonstrates in turn.

The first hypothesis, the one ranked as most likely, fell apart on the first attempt. I ran the same test with one worker, on a dedicated port and in a dedicated Compose project, exactly as I have divided environments into slices since October 2021, and ran it twenty times in a row. It failed. Not once, but six times, and more importantly, only in the final six runs. If it fails without any concurrency, concurrency is not the cause, and the whole attractive story about two workers racing for one port is false. The model put it first because a race with four workers sounds plausible, not because anything in the log points to it.

The third hypothesis died even faster and did not require a run at all. This test is not part of the gate at all: the gate runs unit tests, contracts, and one smoke test, while cart/coupon.spec.ts belongs only to the overnight suite, so there is no green second attempt at the gate masking anything. I told the model explicitly in the prompt that this was an overnight run rather than the gate, and it still put the gate in its hypothesis. In the overnight project itself, retries has been set to zero since the configuration file was created, as the history of playwright.config.ts shows: reruns happen when a person clicks “re-run” in the morning, not when the runner retries the test. Two minutes, one glance, one hypothesis off the list. This is, in fact, the best kind of suggestion I get: not one that is correct, but one that can be ruled out more cheaply than through reproduction.

The second hypothesis got the category right and missed the mechanism completely. Data really was the cause, but not in the way the model described. In my seed data, SUMMER10 is a code with a usage counter set to twenty. I start my local Compose environment from the same snapshot as the persistent overnight environment, so the counter arrived partly used, with fourteen uses remaining. No one consumes it “first” in a race; it simply runs out over successive runs, and nothing resets it. That is why it failed six times at the end rather than randomly in the middle, and why the order of results in my --repeat-each run was more important than the failure itself.

I completed the final step without the model, and that is the point of this post. I asked why the counter runs out specifically in the overnight suite, and I had the answer in my own division into buckets: jobs behind the gate start their dependencies in containers for the duration of the job, exactly as I described in November 2021 when writing about Testcontainers on Azure Pipelines, so each starts with a fresh counter and none runs long enough to chip away at it. The overnight E2E suite, the only place where this test runs, uses a persistent test environment that no one reseeds. The model could not know this because I did not paste it into the chat, and there is no way to infer it from the log. The fix is boring: a discount code is created per test in the data factory, the test cleans up after itself, and the persistent environment’s seed process now has its own job. Twenty consecutive overnight runs were green.

The result from the entire notebook looks like this:

And these are the categories the model targeted:

Of those nine confirmed hypotheses, I would have named six myself within fifteen minutes because these are things that have been breaking in my suite for years. Three were ideas I would not have put first, and that is the entire real value of this exercise. The model did not shorten reproduction by even a minute because reproduction means twenty runs and reading the trace. It expanded the list I take into reproduction, and it did that in twenty seconds.

One more observation matters for anyone tempted to read the order of hypotheses as a ranking. I pasted the same log twice, in two separate conversations, using the same prompt; I did not count this repeat in the notebook. The second time, I got three sensible hypotheses in a different order, and one of them had not appeared in the first conversation. This is not a flaw that can be straightened out with a prompt. It is a property of the tool that must be remembered whenever someone writes that “the model identified the cause.”

Where the model gets it wrong

Six of my sixteen rejected hypotheses were not so much inaccurate as unrelated to my world. They are worth naming because they recur endlessly.

It blames Selenium when the test uses Playwright. Twice I received a lecture about implicitlyWait, ExpectedConditions, and the “known instability of the browser driver” for a log that plainly says locator.click and waiting for element to be visible, enabled and stable. Playwright waits for those three conditions by itself, which is what that entire log section is about. The model sees “browser test, timeout” and reaches for the most common pattern in its training data, where Selenium has a ten-year advantage in volume.

It invents jobs, files, and flags. I was told to “check the Publish E2E step in the e2e-nightly job” and to “set retries in playwright.ci.config.ts.” None of those three names exists in my repository, and the model wrote them without a hint of doubt, as if it had read the repository. This is the same category I caught in April in Copilot’s suggestions during a test code review: the shape is credible, but the specific detail is invented. Once, I was also given an expect.configure() option with a per-assertion timeout that simply does not exist in 1.36.1.

It proposes a sleep. When I forgot to include the sentence banning fixes, I received a package containing page.waitForTimeout(3000) before the click, an increase of the global timeout to sixty seconds, and --workers=1. Each of these three things makes the red result disappear, and none answers why the button was disabled. The third is the worst because it costs wall-clock time across the entire suite and looks like an engineering decision.

It reads intent into a line that is a warning. Once I received an entire hypothesis built on a log line about an unused environment variable that has been in my pipeline for a year and means nothing. The model has no way to distinguish noise that is always present in the log from a signal that appeared today. I can distinguish them because I remember what my green run looks like.

It does not say “I don’t know.” This is the most tiring problem, and it did not disappear in either gpt-3.5-turbo or GPT-4. Asked for three hypotheses, it always gives three, even when the log contains one line and the sensible answer is “paste the trace.” In one run with a network failure on the agent, the third hypothesis was filler, and I could see that immediately. I therefore treat the number in the prompt as a limit, not a promise, and I do not read the third point as equivalent to the first.

The conclusion is the same one I wrote down in March and repeated in April, only at a third layer. The model is good at shape and weak at specifics that depend on a reality it cannot see. An incident category, a list of things to rule out, a reminder about a class of problem I had not considered - yes. A job name, a configuration file name, the existence of an option in my runner version, the history of my environment - no, and I check every one of those four items.

What this does not change in my working practices

The difficult part of triage remains where it was, and I want to write that explicitly so that a year from now it does not look as though I replaced my working practices with a chat in July 2023.

Isolation remains the first step, not the last. The order I wrote down in parallel tests on CI is still there, together with all the boring work: no static state, no fixed port, no shared database between slices. I am not rewriting that post, and I have nothing to change in it because the model does not remove a single one of those requirements. The flake I described here came from exactly the same root as all the earlier ones: data that is shared even though no one declared it as such. The same applies to the place where state lives between tests; the rule from tidying up xUnit and NUnit sounds the same as it did in January 2022: first determine where the state lives, then choose attributes.

Retries remain deliberate. The Polly policies I described in July 2021 are in my integration tests and remain there because they handle transient failures of networks and external services. I do not wrap a retry around a test that lost a race for data, and I do not expand that earlier post with “triage using a model” because these are two different layers. A model that proposes retries as a cure for an exhausted usage counter blurs that exact boundary, and that is a reason to repeat it, not erase it.

I am not building an AI dashboard. I am not connecting a model to CI to read red runs automatically, and I have three reasons, all mundane. First, two identical calls do not have to return the same result, as I demonstrated to myself with two conversations using the same log, so my report would contain text that could not be compared with yesterday’s. Second, a CI log contains things that must not leave the organization, and automation does not redact secrets; it sends what it receives. Third, no one would read it. An automatic comment with three hypotheses under every red run would be new noise, and since September 2022 I have been measuring the percentage of my runs in which something failed without a code change precisely so that there is less noise.

There is also no question of anyone other than a person creating or closing an issue. My overnight run has an owner identified by name in the calendar, and that arrangement from the September strategy does not change by a comma. Someone looks at it in the morning and either creates an issue or deletes the test. The model can be who that person talks to for the first two minutes.

And there is one thing I treat as a boundary, not a transitional stage. gpt-4 has been in the API since 6 July, and I can call it from a script, but the mere existence of access is not a reason to put a model between a red result and my issue. Triage is the activity where I decide whose problem it is, and that is exactly the decision I do not want to hand to something that can confidently provide the name of a job that does not exist.

Summary

After nine days of taking notes, I have a conclusion that fits into one sentence: for me, the model is a faster brainstorming partner during the first division of an incident, and nothing more.

I am breaking this down into three sentences that I want to be able to read a year from now. Of 33 hypotheses, I confirmed 9 through reproduction, but only 3 of them were ideas I would not have proposed myself, and those three are the entire change. I rejected 16 hypotheses after reproduction, including the one the model ranked as most likely in my main example, so the order of the list is not a ranking. The final step, the one that actually explained the incident, came from my knowledge of the difference between the gate and overnight testing, and the model had no chance to take it because that knowledge was not in the log.

What does not follow from this? It does not mean I have a test maintenance tool because I have a browser tab and a pasted log. It does not mean my triage time has decreased because it has not: twenty runs and reading the trace cost as much as they did in June. It does not mean I stopped reading logs because without reading the log, I cannot even redact it for pasting.

I remain the owner of the issue, and that is not false modesty. I put my name under a ticket that goes to the team with the sentence “this is a product bug, not a flake,” and you can only put your name to something that has been reproduced. The model shortened my path to the list of suspects and did not shorten the path from the list to evidence by even a minute. The latter has always been more expensive, and it still is in July 2023.

In a month or two, I will know whether those three non-obvious hypotheses from July are a lasting gain or a nine-day coincidence. Until then, I return with this to the question I ask myself about every new tool: where does it fit in the entire testing process? A faster list of hypotheses does not fix a process where no one looks at the morning report. It only means that the person who does look has something to think about before clicking “re-run.”