All posts

Test documentation from an LLM - October 2023
Test documentation from an LLM - October 2023

Polski

Test documentation from an LLM - October 2023

October 2023: a test plan draft, AC checklists, and a README from a ticket and code. Custom instructions help, but a human publishes. Three deletions.

I have a ticket and a test directory, I want a README, not a keynote

It is 15 October 2023. I have two things on my desk: a ticket with acceptance criteria and a tests/ directory with a few Playwright files. The third thing is missing: documentation that someone else can read before touching the code. A test plan that says what is in scope, an acceptance criteria checklist, and a README that answers one question: how to run it locally.

This is not a new problem. The test process I laid out here in 2019 assumed that someone knew which tests were fast and which ran overnight. Last year’s split between PR, main, and nightly assumed the same. In both cases, that knowledge lived in my head while the repository contained only code. Documentation was always the task I did on Friday afternoon or did not do at all.

This post has one argument, and I want it near the top so it does not get lost in the demo. Today, a language model is good at producing a first draft of test documentation, but only when its source is a ticket or code rather than its memory of a “typical banking application.” When I ask an open question, I get a keynote: four environments, a browser matrix, a risk management section, and no connection to my repository. When I paste the acceptance criteria and file list, I get a draft that I can correct faster than I could write it from scratch.

In January, I wrote here about generating test cases with a model, and this is exactly the same mechanism one level higher: I am not asking for domain knowledge, I am asking it to rewrite material I already have into a prescribed form. The difference is that I verify a test case by running the test. I can verify a document only by reading it, so the gate has to be different.

Input: AC from the ticket, file list, and how to run it

The model receives three things and nothing else. The ticket, the state of the repository, and how to run it.

I invented the ticket for this post from scratch. It contains no number from any system, no client name, and no internal address. That is not decoration for the post but a working rule: I do not paste someone else’s ticket into a prompt.

Synthetic ticket: Password reset through an email link

A user who does not remember their password enters an email address at /reset
and receives a message with a one-time link to the new password form.

Acceptance criteria:
AC1. The form at /reset accepts an email address and always displays the same
     confirmation message, whether or not the account exists.
AC2. The email with the link is sent within 60 seconds for an existing account.
AC3. The token in the link is single-use and expires after 60 minutes.
AC4. The new password has at least 12 characters; the form rejects a password
     identical to the previous one.
AC5. After a successful password change, all active sessions are invalidated.
AC6. Three reset attempts from one IP address within 10 minutes result in
     a 429 response.

The second item is the state of the repository, pasted as a plain listing with no prose description:

$ ls tests/
account-profile.spec.ts
auth-login.spec.ts
auth-logout.spec.ts
password-reset.spec.ts
helpers/mail-stub.ts
helpers/rate-limit.ts

The third is an excerpt from the workflow, so the model knows what runs and when. I do not summarize it in a sentence because a summary is where the truth gets lost:

on:
  pull_request:
  schedule:
    - cron: "0 2 * * *"

jobs:
  pr:
    steps:
      - run: npx playwright test --project=chromium --grep-invert @slow
  nightly:
    if: github.event_name == 'schedule'
    steps:
      - run: npx playwright test --workers=4

There is also a layer I do not paste into every conversation because it has been in my settings since July. Custom instructions arrived in ChatGPT on 20 July, initially in beta for Plus subscribers and later more broadly, and this is the most practical change of the past six months for me. Not because the model became smarter. Because the five sentences I used to repeat in every prompt now apply by default:

You respond in Polish.
You write only from the material provided in this conversation.
Anything absent from the material is marked TODO(human), and you do not guess.
You do not invent environment names, CI jobs, services, or addresses.
You return Markdown with headings and tables, with no introduction or summary.

The effect is measurable and modest. The sentence about guessing reduced the number of invented details in the draft, but did not eliminate them: the three deletions later in this post came from a conversation in which these instructions were active. The formatting sentence works best of the five because a table is a shape, not a fact. The instruction to write in Polish saves me from calques such as “test cases were executed.”

I work in a browser window in the Plus interface. Copilot Chat has been in public beta for individual subscribers since 20 September, so I also have chat in my editor and use it for code, but I assemble documentation where I have persistent instructions and the full history of one conversation. None of this is connected to the pipeline. In this story, the test suite calls nothing over the network.

Three artifacts: plan, checklist, README

I do not ask for “documentation.” I ask for three separate artifacts in three separate messages because each has a different cost of error.

Test plan: scope and out of scope

After my deletions and additions, the first draft looks like this:

# Test plan: password reset through an email link

## Scope

- /reset: email validation and the same message for an existing and
  nonexistent account (AC1)
- sending and delivery time measured against the email stub (AC2)
- token: single use and expiration after 60 minutes (AC3)
- new password: length and rejection of the previous password (AC4)
- session invalidation after the password change (AC5)
- attempt limit from one IP address and a 429 response (AC6)

## Out of scope

- email HTML template and content: TODO(human), owner not established
- delivery through a real email provider, SPF, and DKIM
- authentication through an external provider
- password history beyond the one previous password; AC4 mentions one

## Risks

- AC1 versus AC2: because the message is the same, the test for a nonexistent
  account has no signal in the UI and must inspect the email stub
- AC3: without control over application time, the expiration test means
  waiting 60 minutes, making it a candidate for @slow and the nightly run
- AC6: nightly runs with --workers=4, and the limit applies per IP address;
  parallel workers on one agent see one address

## Levels

- unit: token generator, password validator
- integration: session invalidation, rate-limit counter
- Playwright e2e: AC1, AC2, and AC4 on one path

The “out of scope” section is the most valuable part here, which surprised me in this experiment. I usually wrote a plan as a list of what I would test, while omitted items stayed in my head, where nobody could pull them into a review. The model readily fills this section because it is reasoning from the ticket by negation: the ticket says one previous password, so deeper password history is out of scope.

The third risk is the best thing I got from this conversation, and I would not have come up with it any faster myself. Connecting --workers=4 to a per-IP limit is possible only because I pasted the workflow. If I had written “the tests run in parallel overnight,” I would have received nothing but generalities. The entire difference between a useful draft and one to discard lies in whether the input is source material or a summary of that material.

AC checklist

The second artifact is a table, and here the model is simply a few minutes faster than I am. Six criteria, each with a test, file, and status:

| AC  | What I check                         | Level       | File                   | Status  |
| --- | ------------------------------------ | ----------- | ---------------------- | ------- |
| AC1 | same message, account exists         | e2e         | password-reset.spec.ts | present |
| AC1 | same message, account does not exist | e2e         | password-reset.spec.ts | missing |
| AC2 | email reaches stub within 60 s       | e2e         | password-reset.spec.ts | missing |
| AC3 | token is single-use                  | integration | TODO(human)            | missing |
| AC3 | token invalid after 60 minutes       | integration | TODO(human)            | missing |
| AC4 | password shorter than 12 characters  | unit        | TODO(human)            | missing |
| AC4 | password identical to previous one   | unit        | TODO(human)            | missing |
| AC5 | sessions invalidated after change    | integration | TODO(human)            | missing |
| AC6 | fourth attempt returns 429           | integration | helpers/rate-limit.ts  | missing |

I filled in the “status” column, not the model, and that is a deliberate decision. The model saw the file names but not their contents, so it has no right to claim that anything is covered. The one time I let it guess this column, it returned “present” in five rows based only on the name password-reset.spec.ts. Such a document is worse than no document because it looks reviewed.

Splitting one criterion into two rows is also the model’s work, and it is good. AC1 requires two different tests even though it is one sentence in the ticket. AC3 covers single use and time, two distinct things in one line.

README: how to run it locally

Here the model comes closest to being safe because its source is code rather than a ticket. I paste package.json, playwright.config.ts, and the Compose file, and it turns them into instructions that nobody has to retrieve from a chat channel:

# Tests: password reset

Requirements: Node 18 or newer, Docker with the Compose plugin.
Playwright is pinned to 1.39.0 in this repository.

    npm ci
    npx playwright install --with-deps chromium
    docker compose up -d mail-stub
    npx playwright test tests/password-reset.spec.ts --project=chromium

Tests marked @slow are excluded on pull requests and run in the nightly
job (cron 02:00). Locally: npx playwright test --grep @slow

I give the model the Playwright version explicitly because that is the only way to stop it from guessing. Version 1.39.0 was released on 11 October, four days ago, and just as I pin image tags and model snapshots, I pin the version in the README. A version number in documentation that has drifted away from package-lock.json is the first thing that costs a new person an evening.

Three deletions from the first draft

The plan draft was less than a page long and contained three statements I deleted. All three appeared in a conversation with active custom instructions, so I treat them as the floor, not the ceiling, for the error count.

First deletion: invented environments. The draft contained a section I had not requested:

## Environments

| Environment | Address          | Data                         |
| ----------- | ---------------- | ---------------------------- |
| DEV         | dev.internal     | anonymized production copy   |
| UAT         | uat.internal     | complete data set            |
| PREPROD     | preprod.internal | production copy from 24h ago |

I have a local environment and a container in CI. Nothing else exists, and none of it was in my input. This is not a mistake in one detail. It is the whole mechanism of this post in one place: where the ticket is silent, the model inserts an average from its training material, and the average test document on the internet describes a large company with four environments. I delete the entire section and do not ask for a correction because the correction will return as a smaller version of the same thing.

Second deletion: docker-compose in a version I do not have. The README contained this line:

docker-compose up -d mailhog

Two errors in one line. First, my Compose file has a service called mail-stub, while mailhog is a name the model added because that is what an email stub is often called in its training material. Second, docker-compose with a hyphen is Compose V1, retired this year, while today the plugin is installed and invoked as docker compose. A new person who runs that line gets “command not found” and cannot tell whether their Docker installation or my repository is broken. The correct version is docker compose up -d mail-stub, and that is what goes into the file.

Third deletion: an invented deployment target. This deletion survived two readings of the draft, which makes it the most important one:

## Deployment and post-deployment tests

After merging to main, the pipeline deploys the change to a staging slot in
Azure App Service, and smoke tests run against the slot before the swap.

The workflow I pasted ends at npx playwright test. It contains no deployment, slot, or swap. The statement survived two readings not because it was clever, but because it sounded like a sentence from my world: Azure, a slot, smoke tests after deployment. I have written about all of these here before in a different context. An invented fragment that matches the author’s style is harder to catch than one that does not. It is also more dangerous because such a document enters the next conversation as input material, at which point fiction becomes a source.

Three deletions from less than a page is a balance I accept because correcting the draft is cheaper than writing from scratch. I accept it on one condition: I can verify every line against the ticket or repository. The day I cannot, that balance stops mattering because I do not know how many deletions I missed.

The procedure itself is boring, and that is how it should be. I read the draft beside the input in two windows. Every sentence containing a number, name, or address must have a source in the material. I do not correct sentences without a source; I delete them. At the end, I add what the model could not have known, and I am the author of the commit.

Where I do not use an LLM

Two things remain on my side, and this is not caution for its own sake but a calculation of the cost of error.

The nightly report as the sole source of truth. The model can group similar failures from an overnight run, and that text is easier to read in the morning than a list of one hundred assertions. The problem is what happens next: a summary saying “three flaky, one real” posted to the team channel stops being a summary and becomes a decision nobody has verified. The run report and its trace remain the authoritative document, not a paragraph about them. I can read the summary at my desk to know where to start. I do not publish it as a settled conclusion or put the model into a loop where it reads the result and updates the documentation itself.

A changelog for a client. This text goes outside the company and carries my name. The model does not know what actually reached production because my input does not contain that information, and a sentence such as “password validation was improved” in a document that someone reads as a promise costs more than an evening of writing. To make it useful, I would also have to paste client context into the prompt, and I do not do that. This rule predates language models: my 2021 experiment with the GPT-3 API reached the same conclusion, only about data. Nothing belonging to someone else goes into a prompt: no data, ticket, or name.

A third thing is less obvious: documentation from the model is not a “living specification.” I have already seen the suggestion that a test plan should be generated after every change and treated as the current state of knowledge. That is the same as generating test data on every suite run: paying for something nobody has read and introducing someone else’s nondeterminism where I need one version. A document is living when someone changes it and signs it, not when it is generated.

Summary: a draft from the model, documentation merged like code

After a week, my conclusion fits into one sentence: today, a language model is good at producing a first draft of test documentation as long as its source is a ticket or code and a human publishes it.

I am breaking this down into four sentences that I want to be able to read a year from now. Draft quality depends almost entirely on input quality: acceptance criteria pasted verbatim and a workflow pasted verbatim produce observations I would not have noticed myself, while a prose description of the same material produces generalities. Custom instructions from July are genuinely useful because they move formatting rules and the prohibition on guessing from the prompt into settings, but they are not a gate, and all three deletions in this post came with the instructions enabled. Invented content falls into recurring classes: nonexistent environments, an outdated or distorted command, and an invented deployment target. The most dangerous invention is the one that sounds like my own sentence. The model does not fill in “covered” columns because it has not seen the coverage.

What does not follow from this? It does not mean I have a new tool in the pipeline because all this work happens at my desk in a browser window. It does not mean documentation writes itself because the deletions and additions take as much time as writing the first version used to take; they are simply more pleasant, so I postpone them less often. Nor does it mean the model understands my project: it understands my listing.

The rule I am keeping is short. I merge documentation the way I merge code. The plan, checklist, and README live in the repository beside the tests they describe, appear in a diff, and have one reviewer and one commit author. The model lowers the cost of the first draft and nothing more. Responsibility does not move by a millimeter because you can put your name only to something you have read.

For November, I am leaving myself an adjacent topic that I am not touching today: security. The AC checklist in this post has six rows about functionality and not one about what happens when someone treats the password reset form as an input rather than a feature. Until then, better-looking documentation does not fix my process. It only makes what the ticket failed to say visible sooner.