A paragraph from the night, not a new portal
It is Tuesday, October 15. The nightly run started from the schedule at four in the morning our time, because the 0 2 * * * cron expression uses UTC, and finished before five. An automated message with numbers and links is waiting in the team channel, while the run artifacts hold three Playwright HTML reports, one for each of the three jobs in the nightly matrix. Twelve failed tests in total.
I know what the rest of the morning looks like because I have been doing this for years. I open the first report, scroll, open the second, compare the names with what I remember from yesterday, and after a dozen or so minutes I have a picture of the night in my head that nobody else can see. The team sees a red message and nothing more. I want to close that gap, but not with a new portal full of charts. I want a paragraph for myself and the team: how many failed, what is new, what we already know about, and where I will start.
My thesis for October 2024 is narrow. GPT-4o mini is cheap enough that drafting such a paragraph from an extract of JUnit files costs me a fraction of a cent, and capable enough to organize twelve failures faster than I can open the second report. The quality gate and report publication stay with me: the test sets the color, while I decide what to do about that color and what reaches the team. The model writes a draft. I review it and paste it into a thread or an issue, or discard it completely.
This continues the triage work from July 2023, but it does not repeat it. Back then I had one failed test and asked for three hypotheses that I later checked by reproducing the failure. Here I have an entire run and ask for no hypotheses at all, only order. That post put a microscope to one failure. This one is a map of the night from which I choose where to put the microscope.
Here is the state of the workshop, so that a year from now it will be clear what I mean. The runner is Playwright 1.48.0, released on October 8, a week ago. The draft is written by gpt-4o-mini, pinned to the gpt-4o-mini-2024-07-18 snapshot from launch day, at 15 cents per million input tokens and 60 cents per million output tokens. I have GPT-4o from May 13 in chat and do not use it for this job: the task is to rewrite and group a short extract, while the larger model costs roughly an order of magnitude more per token. o1-preview has been available since September 12 and remains an option, not the pin. It is a preview, has a weekly message limit in Plus, responds slowly, and through the API costs one hundred times more per token than mini, before counting the reasoning tokens that are also billed. Claude 3.5 Sonnet from June 21 is a real competitor, but the choice of provider changes nothing about how I check the draft.
What will not appear here: the model in this post has no authority over anything. It does not touch the gate, issues, or test code, and it does not receive a single access token. There will be no client name or real dump from a nightly run, because nightly logs carry data that is not mine. Nor will there be a review of ready-made dashboards that promise the same thing for a subscription. There will be a synthetic nightly run from my example repository: twelve failures, including three timeouts with similar names and one new assertion on login, together with the prompt, the draft, and three deletions. The underlying application is invented, while the failures, messages, and numbers are exactly what runner 1.48.0 printed.
Input, not model memory
The model does not know my pipeline, and it will not learn it from one sentence in a prompt. It receives an artifact from the run, not a request to remember how things work in our project.
The nightly job is a matrix of three shards with fail-fast: false, built in the same way as in October 2021, except that Playwright now distributes files among the shards itself through --shard. Each shard leaves two artifacts: an HTML report and a file from the built-in JUnit reporter. If I collected blob reports, npx playwright merge-reports could combine them into one HTML report, but a merged report is still a list to click through, not a paragraph. In the morning, I therefore download only the JUnit files and pass them through a script:
gh run download <run-id> --pattern 'junit-*' --dir nightly/2024-10-15
python3 tools/nightly_digest.py nightly/known.txt nightly/2024-10-15The script is 26 lines long and does not call any model:
import sys
import xml.etree.ElementTree as ET
from pathlib import Path
known = set(Path(sys.argv[1]).read_text(encoding="utf-8").splitlines())
reports = sorted(Path(sys.argv[2]).glob("junit-*/junit.xml"))
rows = []
for report in reports:
shard = report.parent.name.removeprefix("junit-")
for case in ET.parse(report).iter("testcase"):
failure = case.find("failure")
if failure is None:
continue
name = f'{case.get("classname")} › {case.get("name")}'
lines = [line.strip() for line in (failure.text or "").splitlines()]
error = next((line for line in lines if "Error" in line), "?")
diff = [line for line in lines if line.startswith(("Expected", "Received"))]
rows.append(("KNOWN" if name in known else "NEW", shard, name, error, diff[:2]))
new = sum(1 for row in rows if row[0] == "NEW")
print(f"failures: {len(rows)}, new: {new}, known: {len(rows) - new}, shards: {len(reports)}")
for label, shard, name, error, diff in rows:
print(f"{label:<5} {shard}/{len(reports)} {name}")
print(f" {error}")
if diff:
print(f" {' | '.join(diff)}")It does three things. From each file, it takes only cases that have a failure element. From the failure content, it keeps the first line containing the word Error and a pair of Expected and Received lines if the runner printed them. It compares the full test name with the list in nightly/known.txt and adds either the NEW or KNOWN label. I maintain that list manually, and every name on it has an issue or scheduled triage behind it. What is known is my memory, not the model’s memory, which is why it lives in a file next to the script.
The extract from the night of October 15 looks like this:
failures: 12, new: 1, known: 11, shards: 3
KNOWN 1/3 account/orders.spec.ts › orders › recent orders list
Error: Timed out 5000ms waiting for expect(locator).toHaveCount(expected)
Expected: 3 | Received: 0
KNOWN 1/3 account/orders.spec.ts › orders › filter by status
Error: Timed out 5000ms waiting for expect(locator).toHaveCount(expected)
Expected: 1 | Received: 0
KNOWN 1/3 account/orders.spec.ts › orders › order details
Error: Timed out 5000ms waiting for expect(locator).toBeVisible()
Expected: visible | Received: <element(s) not found>
KNOWN 1/3 account/orders.spec.ts › orders › repeat order
Error: Timed out 5000ms waiting for expect(locator).toBeVisible()
Expected: visible | Received: <element(s) not found>
KNOWN 1/3 account/returns.spec.ts › returns › return form
Error: Timed out 5000ms waiting for expect(locator).toBeVisible()
Expected: visible | Received: <element(s) not found>
KNOWN 1/3 account/returns.spec.ts › returns › return status
Error: Timed out 5000ms waiting for expect(locator).toHaveCount(expected)
Expected: 1 | Received: 0
KNOWN 2/3 account/reviews.spec.ts › reviews › submit a review
Error: Timed out 5000ms waiting for expect(locator).toBeVisible()
Expected: visible | Received: <element(s) not found>
KNOWN 2/3 account/reviews.spec.ts › reviews › reviews list
Error: Timed out 5000ms waiting for expect(locator).toHaveCount(expected)
Expected: 2 | Received: 0
NEW 2/3 auth/login.spec.ts › login › nonexistent email gets the common message
Error: Timed out 5000ms waiting for expect(locator).toHaveText(expected)
Expected string: "Invalid email or password" | Received string: "Account not found"
KNOWN 3/3 checkout/delivery.spec.ts › delivery › courier
TimeoutError: locator.check: Timeout 15000ms exceeded.
KNOWN 3/3 checkout/delivery.spec.ts › delivery › parcel locker
TimeoutError: locator.check: Timeout 15000ms exceeded.
KNOWN 3/3 checkout/delivery.spec.ts › delivery › pickup point
TimeoutError: locator.check: Timeout 15000ms exceeded.The three XML files from that night total about 17 kilobytes, while the failure contents alone contain 270 non-empty lines. The extract has 34. The rest is the call log, which repeats the same pair of lines nine times for the login assertion, code frames, and the agent path attached to every failure.
This also answers the redaction work that I did with scissors in July last year. At the time, I cut tokens, Authorization headers, internal service addresses, and anything that could identify the project out of the log. Today, most of those things have no way to get in, because the script reads three fields and nothing else. It does not read system-out or system-err, which receive everything the test or application printed to the console. It does not read the call log with addresses and selectors. It does not read the agent path, which begins with /home/runner/work/ on a GitHub-hosted runner and continues with the repository name. Redaction moved from scissors to construction: what the script does not read cannot appear in the extract.
Construction does not release me from reading, however. The Expected and Received lines contain what was on the page, and the page can contain an email address, order number, or surname from test data. The extract has 34 lines, so I read all of it before sending anything. In somebody else’s project, the rule from the triage post applies without a single amendment: pasting somebody else’s artifact into an external service is its owner’s decision, not mine, and the extract goes to the model using a key controlled by that owner.
One sentence that does not appear in any JUnit file is added to the extract: where the run came from. In September 2022 I divided tests into three buckets, and that division does all the work here. A red test on a pull request is the change author’s problem before merging. A red test at night is the nightly run owner’s problem to identify in the morning. The same twelve failures mean different things in those two places, and without that sentence the model guesses. In July last year, it guessed even with that sentence and gave me a hypothesis about the gate for a test that does not run at the gate at all. That is why I state it directly in the prompt: schedule, not pull request, three shards, and no other jobs.
What the draft should contain
The draft format is the most important decision in this entire exercise, because it is the only place where I can impose anything on the model.
The count goes at the top: failures, new, known. It comes first because it is the easiest thing to check mechanically, which is why the model must copy it from the script rather than calculate it. New failures follow, each separately, even if there is only one. Known failures come next, one line per group, because the details are in the issues. Then there are no more than three clusters and no more than three things to check in the morning, because nobody will check anything from a list of ten. The final section is the one I missed most in July last year: what I do not know. The model did not say “I don’t know” then, and the response format left no place for it. When the section is in the template, the model fills it in.
The clusters have three names, each describing a symptom rather than a cause. A timeout is a failure where the runner did not see an element in time to perform an action on it. Data is a failure where there is nothing to check because a list is empty or an element is missing from the page. The name sounds like a cause, but in the draft it means only that. An assertion is a failure where the element exists but has different content than the test expects. In July 2023, I divided failures into four cause buckets: test, data, infrastructure, product. That division is useful for one failure that I am about to reproduce. For a map of the night, I divide by what the runner saw because that can be checked in the XML. This is why there is no “product” bucket in the draft: product is a verdict, and a verdict requires reproduction. There is no verdict in the other direction either, because “it is all the environment” is the same kind of verdict, only cheaper.
I keep the prompt in docs/nightly-prompt.md, next to docs/triage-prompt.md from July last year, and send it as a system message. The extract is the second message:
This is an extract from a nightly run started by the schedule, not a pull request.
Runner: Playwright 1.48.0. The nightly job has three shards: nightly (1),
nightly (2), and nightly (3). There are no other jobs in this run.
The script calculated the numbers in the header and the NEW / KNOWN labels.
Copy them. Do not calculate them again.
Write a report draft in English, in this order:
1. Count: failures, new, known.
2. New: every test with the NEW label separately, even if there is only one.
3. Known: one line per group.
4. Clusters, no more than three, with the number of tests and shared message:
timeout - the runner did not see an element in time to act on it,
data - there is nothing to check, a list is empty or an element is missing,
assertion - the element exists but has different content.
5. To check in the morning: no more than three items.
6. What I do not know: what cannot be determined from this extract.
Do not say that anything is a product bug. Do not suggest fixes
or rerunning the tests. Do not mention jobs, files, or tests
that are not in the extract.Here is the draft from the night of October 15, without any edits from me:
Nightly 15 Oct, scheduled run (not PR), Playwright 1.48.0
Count: 12 failures, new: 1, known: 11.
New:
- auth/login.spec.ts › login › nonexistent email gets the common
message: instead of the expected "Invalid email or password",
the page shows "Account not found".
Known:
- 8 account tests (orders, returns, reviews): empty lists or missing elements.
- 3 delivery tests (courier, parcel locker, pickup point): timeout while selecting an option.
Clusters:
1. Data (8): Received: 0 or <element(s) not found> on account pages,
shards 1/3 and 2/3.
2. Timeout (3): locator.check exceeds 15000 ms on delivery options,
shard 3/3. This is typical WebDriver instability with slowly
loading elements; increasing the implicit wait will help.
3. Assertion (1): different login message text, shard 2/3.
The nightly-api job completed without errors, so the problem does not affect
the API layer.
To check in the morning:
- whether the test account has orders, returns, and reviews,
- a trace from one of the delivery timeouts,
- where the login message change came from.
What I do not know: whether the timeouts come from the application or the CI
agent, and whether the message change was intentional.
Summary: the night looks like instability in the test environment.
After replenishing the data, the entire run should be rerun.I will start with what is good about this draft, because there is more good than bad. Mini combined eight failures from three files, two shards, and two different assertions into one cluster and named it after the symptom rather than the file. It brought the new test to the top, separately, with a sentence that says exactly what is visible in the extract. The list of things to check in the morning is my list, in my order. The “what I do not know” section asks two questions that I really cannot answer from the extract alone. Clicking through the same information in three HTML reports takes me a dozen or so minutes, while the draft is ready before I close the first report.
I delete three sentences from this draft and take each one apart later, in the section about how the model lies. The draft itself is a working document for me and does not leave my window. What reaches the channel must fit in a Slack thread without scrolling.
Where the human publishes
After the deletions, the draft is correct and incomplete. I add two things that do not exist in any extract.
The first is the history of main. A login form message change was merged there yesterday afternoon. A nonexistent address is not a pull request gate scenario. It runs at night with the full E2E suite, so the change passed on green and only the nightly run saw it. That is precisely what the third bucket is for.
The second is the criterion this test protects. It belongs to the login story for which I drafted cases with ChatGPT in January last year, before its test suite existed. The criterion says that invalid credentials display one message without revealing which field is wrong, while the list of cases I left at the time states explicitly that an email address that does not exist in the system gets the same message, without revealing whether the account exists. “Account not found” reveals exactly that. Neither the model nor I decide whether this is a bug or an intentional criterion change. The change author and the story owner decide. My role ends with the question and an issue containing the trace. From there, it is triage with the microscope.
The known failures have their own issues, and I add only their status. Eight account tests have been failing since yesterday because the persistent nightly environment did not receive its seed data. The three timeouts on delivery options return every few nights, and in the history of nightly runs this is the third red night out of the last seven.
Here is the paragraph I pasted into the thread under the automated message from the night:
Nightly 15 Oct: 12 failures, 1 new.
New: login › nonexistent email gets the common message.
Instead of "Invalid email or password", the page shows "Account not found",
while the criterion requires one message without revealing whether the account
exists. A login message change was merged into main yesterday afternoon.
The question for the author and the trace are in the issue.
Known: 8 account tests without data (seed, issue from 14 Oct)
and 3 timeouts on delivery options (third night out of the last seven,
triage this week).
I am not rerunning anything. The HTML and JUnit reports are in the run artifacts.What remains from the model is the numbers, the name of the new test, both clusters of known failures, and the order. What comes from me is yesterday’s merge, the criterion, issue status, the seven-night series, and the final sentence. I can put my name under a paragraph like this because I checked every sentence against the extract, the XML, or the repository history.
Now for what this paragraph does not do, because this is the entire boundary. The nightly run’s color is set by the runner exit code. Playwright exits with an error when at least one test fails, the job is red, and nothing I write in a thread changes that. The nightly run is not a required check and blocks no merge, while the pull request gate is green or red because of the runner, not the paragraph. No pipeline step reads what the model wrote. Nor does the model have a token for anything: not Slack, not the repository, not issues.
Publication mechanics remain mechanics. For Cypress results in Azure DevOps, the test step got continueOnError: true so that result publication could run at all, while failTaskOnFailedTests: true transferred the red status to the publication step. Since then, I have known that configuration sets the color, not a description. I have been sending a Slack message after tests since 2020 through a step that always runs: back then with condition: always() in Azure DevOps, today with if: always() in Actions. It goes out with report links regardless of the result and remains the first signal in the morning. The paragraph replaces neither publication nor the message. It goes into the thread under the automated message, so the numbers from the mechanism and my sentences sit next to each other. If they differ, the paragraph is wrong.
The paragraph has three possible destinations, and I choose among them. The thread, when something new happened during the night. An issue, when a new failure needs an owner, in which case the issue gets a sentence from the paragraph, the trace, and a report link, not the entire draft. The trash, when nothing is new. The automated message then says everything that needs saying, while a daily “11 known, 0 new” paragraph would be exactly the noise I guarded against in July last year by not putting the model into CI.
I need to address honestly what I wrote a year ago about test documentation with an LLM. I wrote then that once a nightly summary is posted to the team channel, it stops being a summary and becomes a decision that nobody verified, and that I do not publish such a summary as an agreed conclusion. I still stand by that. What reaches the channel is not the model’s summary, but my paragraph: checked sentence by sentence, with two things the model could not have known and a reference to the reports that remain the authoritative document. The difference is not who wrote the first version, but who checked it and who signs it.
The draft did not get its own job either. In June, a model got such a job on pull requests in the example repository, in an advisory role and without any right to the gate. That makes sense there because the author, already waiting for review, reads the comment under the diff. A job after the nightly run that posts its own draft to the channel would publish precisely an unverified summary. A job that only puts the draft into the artifacts would produce unrequested text every night and keep an API key in the pipeline secrets for no reason.
The draft is therefore created in the morning on my laptop, through Chat Completions, with temperature set to zero. The input, meaning the prompt and extract, is about 950 tokens, and the draft is about 420, so at mini’s prices the whole morning costs just under four hundredths of a cent. Zero temperature does not turn the model into a function: two calls with the same extract do not have to return the same text, so I compare extracts, not drafts. I could assemble the same thing as a custom GPT or an assistant with a file, because both have been available since DevDay on November 6 last year, but rewriting 34 lines needs neither memory nor files.
When the model lies
The procedure is the same one I described a year ago for documentation, and it is deliberately boring. The draft in one window, the extract in another. Every sentence with a number or with the name of a job, file, or test must have a source in the extract. I do not correct sentences without a source. I delete them. When something raises a doubt, I open the raw XML or HTML report because the extract is also only a summary.
First deletion: an invented job. “The nightly-api job completed without errors, so the problem does not affect the API layer.” There is no nightly-api job in this run, and the prompt said so directly in a single sentence. My API tests run on main after a merge, in the second bucket from 2022, not at night. This is the same class as e2e-nightly from July 2023: a plausible shape, a fabricated detail. This time it is worse, however, because the invented job is not decoration but a premise. The sentence rules out the API layer on the basis of a job that does not exist, while with eight failures caused by empty lists on account pages, the API would be the first place checked by anyone who does not know about yesterday’s issue. Anyone who reads this sentence on the channel will start looking elsewhere.
Second deletion: WebDriver in a Playwright report. “This is typical WebDriver instability with slowly loading elements; increasing the implicit wait will help.” One sentence contains two errors. The first is the tool. This run has no WebDriver, while implicit wait is a Selenium concept that Playwright does not need because it waits before every action itself. The prompt included the runner and its version, and that did not help, just as the first error line containing locator.click did not help in July last year. The second error is the fix, which I prohibited. In the raw XML, the call log for the courier has one line, waiting for getByRole('radio', { name: 'Courier' }), and the parcel locker and pickup point logs look the same, with different names. For fifteen seconds, none of the three option controls appeared on the page. Waiting longer would produce the same result later or conceal a slow page that somebody should see. In July last year, such advice stayed in my chat window. In a channel paragraph, somebody could simply follow it.
Third deletion: “the night looks like environment instability.” The draft’s final section is called “Summary”, and I did not ask for it. The prompt has six points, none requesting a conclusion. The model added one on its own, flattened the night into one bucket in two sentences, and recommended rerunning the entire run, which I explicitly prohibited. This summary contradicts three things at once. It contradicts the “New” section in the same draft because different login message text is not instability of any kind. It contradicts the “What I do not know” section, where the model itself wrote that it did not know whether the timeouts came from the application or the agent. And it contradicts the data. A test against an empty orders list is not unstable. It fails every time until somebody seeds the account, so a rerun would produce the same eight red tests and the same login assertion. At most, it might turn the timeouts green and make the night look better than it was. In September 2022 I described the moment when the gate becomes a formality: someone clicks “re-run” without reading the log. This summary offers that click to the entire team, and it is the easiest part of the whole draft to copy to the channel because it sounds like a conclusion.
This draft does not contain a fourth failure mode, and the script, not the model, deserves the credit. The first script version did not add labels, while the first prompt version told the model to separate known failures from new ones by itself. For comparison, I ran that version against the same night’s extract with the labels removed and got one cluster for nine tests:
Assertions (9): lists and elements on account pages are empty or invisible
(toHaveCount, toBeVisible, toHaveText), probably because data is missing.The login test was the ninth member of that cluster and appeared nowhere else. The model had no way to know what was known, so it grouped by the first error line because nine of the twelve failures start with Error: Timed out 5000ms waiting for expect(locator), rather than by what the Expected and Received pair says. A cluster with an explanation is the best place to hide a new failure: it looks organized while swallowing the one piece of information for which someone opens the report in the morning. Since then, the script has determined what is new, while the prompt requires every test with the NEW label to be listed separately, even if there is only one.
After the deletions, I run four checks that require no thought, which is why I always run them. The numbers in the draft must equal the extract header. Every NEW line from the extract must appear in the “New” section with the test name. Every job, file, and test mentioned in the draft must appear in either the extract or the prompt. The fourth check concerns the script, not the model: the sum of the failures attributes on the testsuites elements in the three files must equal the failure count in the extract. That night they were 6, 3, and 3, for a total of 12, the same as the header. A script that loses a failure is more dangerous than a model that invents one because nobody reads its output twice.
A better model would not change this procedure. I did not check whether GPT-4o or Claude 3.5 Sonnet would have needed fewer deletions that night, and that is not the question, because each would go through the same four checks. The model changes how much I delete. It does not change the fact that I read.
Summary
In October 2024, my nightly report with a model is a cheap draft from GPT-4o mini, reviewed like a pull request: sentence by sentence, with a source in the extract. The gate stays with the test, not the chat.
Three sentences I want to be able to read a year from now. A 26-line script turns three nightly JUnit files into a 34-line extract, counts failures and labels new ones itself, and the model receives only what the script read. gpt-4o-mini-2024-07-18 turns that into a draft for just under four hundredths of a cent and is good at what amounts to organization and weak at what amounts to a conclusion: from a synthetic night with twelve failures, I deleted an invented job, WebDriver with advice about waiting, and a summary that blames the night on the environment and tells me to rerun it. What goes to the channel is my paragraph with two things the model could not know, meaning yesterday’s merge and the criterion, while the runner sets the run’s color.
What does not follow from this: it does not mean the model entered my nightly pipeline, because the draft is created on a laptop in the morning and no pipeline step reads it. It does not mean triage is unnecessary, because the paragraph ends with an issue that somebody must reproduce, while the map of the night only shows where to put the microscope. Nor does it mean that I save much time. The draft replaced clicking through three reports, but reading the draft alongside the extract consumes roughly half of what I saved, exactly as with test cases in January 2023.
I measure one thing and record it next to every paragraph: how many sentences I deleted. If deleting starts to take longer than writing the paragraph from scratch, the model draft leaves the process, while the extract stays because the extract works without the model. I take the numbers for the 2022 metric, the percentage of nightly runs in which something failed without a code change, from the extracts, not the paragraphs.
In the 2023 summary, I wanted the answer to “where is AI in this process” to fit into one sentence with the verb “review.” For the night, that sentence is: the model drafts a paragraph from the extract, and I review, supplement, publish, and sign it. A cheaper draft does not make somebody look at the nightly run in the morning. It only helps the person who does look move faster from three reports to one paragraph they can put their name under.