A paragraph from the night, not a new portal
It is 15 October. In the morning I have Playwright HTML and three nightly jobs. I want a paragraph for myself and the team, not an “AI test intelligence” dashboard. The model writes a draft. I paste it onto a channel, into a ticket, or I strike it out.
My thesis for October 2024 is narrow. GPT-4o mini is cheap enough to turn XML or a failed list into a sketch: how many failed, which clusters, what I would check in the morning. The quality gate and publication stay with me. In July 2023 I took one log and hypotheses for one failure. Today I take the whole run. That post is a microscope. This one is a map of the night.
Workshop state. The runner is Playwright 1.48.0 from 8 October. For logs I pin gpt-4o-mini from 18 July; GPT-4o from 13 May stays when I want a wider context. I have had o1-preview since 12 September as an option. It is not the nightly pin: slow, expensive, preview. Claude 3.5 Sonnet from 21 June stays a competitor in chat. I am not pinning an upgrade or computer use here.
What will not be here. There will be no “AI owns the quality gate”. There will be no auto-merge and no automatic ticket closing. There will be no nightly dumps with PII. There will be no vendor review. There will be a synthetic nightly: twelve fails, three similar timeouts, one new login assert.
Input, not the model’s memory
I paste an artifact. I do not ask the model to “remember our pipeline”. The input is JUnit or a short list of names, plus one sentence: PR or scheduled. In September 2022 those two triggers mean different things. A model that does not get that will fold the gate into the night.
Secret redaction is the same as in triage. A token, a connection string, a name from a log do not go into the prompt. I keep the test name, the timing, the assertion message. In 2023 the model wrote docs from a ticket before the code existed. Here it gets an artifact after the run. That is not the same prompt.
Job parallelism from October 2021 stays in YAML. I do not ask the model to guess which matrix cell failed if that is not in the XML.
The prompt I started this week with was deliberately narrow:
This is JUnit from nightly, not from a PR.
Count the fails. Make three clusters. Separate a known timeout from a new fail.
Do not write that this is a product bug. Do not invent jobs that are not on the list.
Three paragraphs.What belongs in the sketch
A count. New versus a flake I already know. Three clusters: timeout, data, assertion. One sentence about what I do not know. No verdict that “this is a product bug”. No “it is all the environment”.
The sketch from the synthetic night looked like this: twelve reds. Nine names from checkout.spec fail on TimeoutError at the same step. Two are toHaveText on a date banner. One is new: login.spec and toHaveAccessibleName. That is a map I can read with coffee. That is not a merge.
In January 2023 the model sketched cases before a suite existed. Today it sketches after the suite already failed. Still a sketch.
Where a human publishes
The required check stays green or red from the runner. An AI report is not a gate. I decide whether the paragraph goes to Slack, into a ticket, or into the bin.
Publishing results in CI stays the mechanism I had with Cypress and with Azure DevOps: artifact, card, channel. The model does not replace publish. It adds a paragraph when I paste it.
In the 2023 recap human review was the condition for AI on the team. In October 2024 I have not changed that. A draft from mini is not the owner of quality.
Cursor in this workshop is not a reporting product. Sometimes I paste a summary into chat in the IDE. I still publish.
When the model lies
Three strikes from this nightly.
Strike one: an invented job. The sketch wrote that api-contract failed. There is no such suite in the XML. I compare with the raw file. If the name is not on the list, the sentence goes.
Strike two: Playwright mixed up with Selenium. Mini generalises “browser tests” and inserts WebDriverWait. The runner is 1.48.0. The message from Playwright HTML stays the source, not the model’s vocabulary.
Strike three: “it is all environment flake”. Nine checkout timeouts may be the environment. The new login fail is not. A model that flattens the night into one bucket is worse than raw JUnit. In July 2023 the same temptation sat on one test. On twelve it is more dangerous.
I do not pin o1-preview. Preview, limits, cost on nightly. If it enters later, it enters as a second sketch, not as a gate.
Summary
In October 2024 a cheap draft from 4o mini is a review, like a PR review. The gate stays on the test, not on the chat.
Three sentences for a year. Mini assembles a map of the night from XML it did not invent. I strike the invented job and I publish or I do not. The required check does not read the paragraph.
What does not follow. It does not follow that I delete 2023 triage. It does not follow that 1.48.1 already sits here. It does not follow that the model closes a ticket.
Maturity condition: two nightlies in a row the sketch matches JUnit on the count and does not shout “environment” on a new assert.