An internal AI QA assistant - a practice, not a product (November 2024)

A narrow helper, not a launch

It is Friday, 15 November. I have had GPTs and the Assistants API at hand for a year, because OpenAI showed both at DevDay on 6 November 2023. In the 2023 recap I recorded that as a headline of the year, not as a process: my November work produced neither a single GPT nor a single line calling Assistants. The week and a bit between the keynote and my November post was enough, at most, to summarize the slides, so I postponed both topics until 2024. A year has passed, and today I am closing this item from the list of things I deliberately put aside.

I am assembling a narrow assistant of my own for one class of questions: how we run and read tests here. How to run the login E2E tests locally. Which job runs the load tests. Whether a red nightly blocks a merge. I ask myself these questions when I return to the example repository after two weeks spent on something else, and anyone opening this repository for the first time would ask them. “Internal” here means mine, set up alongside my repository, not deployed at somebody’s company.

The thesis is narrow, and I want it at the top so it does not get lost in the demo. In November 2024, an internal QA assistant is a narrow GPT or an Assistants thread over a synthetic corpus with three parts: a summary of the test process, locator conventions, and a description of how to read the nightly run. In my case, it is a GPT with an instruction and three files that I could publish. This is DevDay prior art, not my invention. The assistant’s answer is a draft that I review against the file, just as I would review a pull request. The assistant does not run the suite, is not a quality gate, and has no token for anything.

A month ago, when working on the draft from a nightly run, I deliberately used neither a GPT nor an assistant with files. The input was an extract from one night, different every morning, and rewriting it required neither memory nor files. Here it is the opposite. The source remains fixed, the same three files, and only the question changes. That is the entire reason a GPT with files makes sense here.

The state of the workshop, so that a year from now it is clear what I mean. The runner is Playwright 1.48.2 from 25 October, and the projects in the example repository remain on .NET 8, although .NET 9 was released on 12 November, three days ago. It remains a headline: I do not rewrite either the stack or the assistant’s corpus in three days. My GPT in ChatGPT runs on GPT-4o and, unlike the API where I pinned gpt-4o-mini-2024-07-18 in October, it cannot be pinned to a snapshot. o1-preview from 12 September is an option, not a pin: it launched without file uploads, and a corpus assistant without files is just a regular chat. On 22 October, Anthropic showed the updated Claude 3.5 Sonnet and computer use in public beta, meaning a model that looks at the screen and clicks. That is a headline, not this post: my assistant has neither a mouse nor a terminal. The Cursor editor is still not my daily tool, and no editor is the QA assistant as defined in this post.

What will not be here. There will be no client names, Jira screenshots, or Confluence pages. There will be no product launch, assistant brand, or price list. There will be no comparison of vector databases, because my files fit on two pages. There will be a synthetic corpus of three files, an instruction, three reviewed questions, and one trap question about a job that does not exist.

Prior art, not an invention

At DevDay, OpenAI showed the same thing in two packages. GPTs are versions of ChatGPT assembled without code from instructions, additional knowledge in uploaded files, and capabilities such as web browsing, image generation, or data analysis. You can make them for yourself, just for your company, or for everyone. The Assistants API is the other package, for developers: an assistant has instructions, knowledge, and tools, namely Code Interpreter, Retrieval, and function calling, while OpenAI keeps a persistent conversation thread on its side. The announcement says directly that the API is built on the same capabilities as GPTs. GPT-4 Turbo was released on the same day, initially as a preview, with a 128,000-token context window.

The GPTs announcement contains a sentence that described me a year ago: many advanced users maintain a list of carefully crafted prompts and instruction sets and manually copy them into ChatGPT. That was my 2023. In January it ended with a file containing three paragraphs of context that I pasted at the beginning of every conversation. In February a template made of four blocks stayed in the notes repository. In July, three paragraphs sat in docs/triage-prompt.md, and I wrote that I had no place where I could set that context once and for all. In October, I moved five sentences to custom instructions, available since 20 July 2023.

A GPT does not add a new category to this. It adds scope. Custom instructions apply to every conversation, so they had to be general: language, format, and no guessing. A GPT instruction applies to one subject and has files beside it, so it can be specific: answer only from these three files.

What did 2024 bring? GPT-4o from 13 May matches GPT-4 Turbo on English text and code, is markedly better in other languages, and in the API is twice as fast and half the price. With a corpus in Polish, that matters. GPT-4o mini from 18 July costs 15 cents per million input tokens and 60 cents per million output tokens, compared with 10 and 30 dollars per million for GPT-4 Turbo from DevDay. A mini token therefore costs dozens of times less than a GPT-4 Turbo token did a year ago. This is a change in cost and model, not a new category, and I have nothing here to announce as a 2024 invention.

Why a GPT in ChatGPT rather than the Assistants API? The Assistants API is the same category and, since July, can run on 4o mini. I choose a GPT for two reasons. The first is prosaic: I need no key, script, or thread handling, and I pay for ChatGPT Plus anyway. The second is more important. A script with a key is one step away from a job in CI, and I see no place in the pipeline for this assistant. In June, the model got a job on pull requests with an advisory voice, because the author, who is already waiting for a review, reads the comment under the diff. A question about the process has neither a diff nor a trigger. A human asks it when they need it.

One feature that DevDay presented as an advantage works against me here. Both a persistent thread in Assistants and regular conversation history in ChatGPT keep previous answers in context. I want every answer to rest on the files, not on what the model said two questions earlier. That is why I ask each question in a new conversation.

What I put in

The input rule is the same as it has been since February 2023: only material I could publish on the blog enters the conversation. The corpus is therefore synthetic. It contains no tickets, names, host addresses, or anything that could identify a project.

The files live in the example repository under docs/asystent/, with the instruction beside them in docs/asystent/instrukcja.txt. Every change goes through a commit with a diff. A year ago, when writing about test documentation with an LLM, I recorded that I merge documentation the same way I merge code, and this corpus is documentation. The GPT receives a copy. The second line of every file is the version date, and after every change I upload the files to the GPT again. Every sentence occupies a separate line so that a quote from an answer can be found with one grep -F.

The first file, proces.md, is a summary of the test process I described in 2019 and the buckets from the division of tests by trigger from September 2022, together with two k6 jobs added in September. This is the version I uploaded on Wednesday:

# Test process (summary)
Version: 2024-11-13

Fast feedback on a code change is the main aspect of building quality.
Ten tests that everyone can count on are better than three hundred that nobody trusts.
Unit and acceptance tests are automated; exploration is manual.

## Buckets
Pull request: job pr-gate, the only required check.
pr-gate runs the build, unit tests, consumer contracts, and one smoke test that checks whether the application starts.
main after merge: job integration (containers, provider contracts) and job perf-smoke (k6, smoke profile).
Night: job nightly (E2E in Playwright, three shards) and job load (k6, load and browser).
Jobs integration, perf-smoke, nightly, and load are not required.
A red nightly does not block a merge.
A new failure from the night gets an issue with an owner on the same day.

## Locally
Required: .NET 8 SDK, Node 20, Docker with Compose, Playwright 1.48.2 pinned in package.json.
dotnet test tests/Unit
docker compose -f docker-compose.e2e.yml up -d
npx playwright install --with-deps chromium
npx playwright test auth/login.spec.ts --project=chromium

The three sentences at the top are from 2019, and I kept them deliberately even though they answer none of the questions in this post. This file is also read by a person opening the repository for the first time, and that person should first learn why this process exists and only then how to run it.

The second file, konwencje.md, takes its structure from the 2021 page object refactor. That post was about C# and Selenium and deliberately left selectors aside, so I add the locator rule only here, together with the data-testid convention:

# E2E conventions
Version: 2024-11-13

A page object describes a fragment of the interface, not a URL: a header, a form, a table row.
Pages only compose components.
Locators live in components, never in the spec.
A page object returns state, and assertions stay in the test.
The component waits for elements; the test has no waitForTimeout.

## Locators
First getByRole with an accessible name, then getByLabel for form fields.
Use getByTestId only when the element cannot be found by role and name or when the test checks the accessible name itself.
data-testid has the form area-element in kebab-case, for example login-email.
A new data-testid is added to the application code, in a pull request, together with a developer.
A test does not assume a data-testid that is absent from the DOM.

The last line is equally for humans and models. I have crossed out an invented data-testid more than once this year.

The third file, nightly.md, compresses the October practice for reading the night into seven sentences:

# How to read a nightly run
Version: 2024-11-13

A nightly run consists of job nightly in three shards and job load.
In the morning, start with an extract from the script: the number of failures, new and known.
The script counts the numbers, not the model.
Known failures are in nightly/znane.txt, and each has an issue or a scheduled triage.
Clusters in the draft describe a symptom: timeout, data, or assertion, not a verdict.
The model's draft is a working copy; the paragraph sent to the channel is signed by the owner of the night.
Rerun only after reading the log.

Together, this is less than two pages. I do not put my own vector database under it or compare strategies for dividing text into chunks, because with two pages that would be engineering for engineering’s sake. My problem is simpler: whether the answer I receive rests on these two pages.

The instruction uses the same four blocks from February 2023, namely role, context, constraints, and format, only moved from the conversation window to the GPT settings:

Role: you answer questions about how we run and read tests
in this repository. You are not a code reviewer or a process adviser.

Context: everything you know about this repository is in the files
proces.md, konwencje.md, and nightly.md. The files are synthetic.

Constraints:
- Answer only on the basis of the files, without general knowledge.
- Give names of jobs, files, commands, and flags only when
  they appear in the files verbatim.
- If the files do not contain the answer, write: "I don't know. The files
  do not contain this information." and add nothing else.
- Do not assess whether a specific change can enter main.
- Do not explain how this is done in other teams.

Format: no more than five sentences. Under the answer, for every claim:
Source: <file> "<sentence from the file, verbatim>"

In the GPT settings, I disable all three capabilities: web browsing, image generation, and data analysis. I add no Actions, and the GPT is shared only with me. As the creator of a GPT with files, I can also decide whether conversations with it may be used to improve the models, and I turn that off.

What the assistant is not allowed to do

The assistant has no hands. Without data analysis, meaning Code Interpreter, it cannot execute any code. Without Actions, it cannot call any API, so it cannot start a workflow, rerun a job, comment on a pull request, or open an issue. It also has no token. It can only answer with text, and that is how it should stay.

I do not run dotnet test from chat, and I do not turn chat into a gate. Even if I enabled data analysis and uploaded the project, a run in the sandbox would not be my run: a different machine, different tools, without my containers and without my secrets. The gate is the exit code from pr-gate in GitHub Actions, and nothing the model writes changes it. For the same reason, I do not paste a run result into the conversation and ask, “may I merge?” The assistant can remind me which check is required. It cannot tell me that mine passed.

I do not put secrets in, including “just for a moment”. No .env, connection strings, tokens, real hosts, or test account data. In last year’s introduction to security testing, two hard rules were: only my own environment and only accounts I created myself, with synthetic data. For the assistant itself, I also apply the question from A01, meaning Broken Access Control: who can see this resource. The GPT is shared only with me, but I plan as if anyone who gets the link could extract the instruction and files from it. Only material I could publish here on the blog goes inside.

A hallucinated job means rejection, without discussion. During triage in July last year, I received a Publish E2E step in job e2e-nightly and retries in playwright.ci.config.ts, three names that do not exist in my setup. In October 2023, a documentation draft invented three environments and an Azure slot. In October this year, a draft from the night named a job that was not in the run. The prohibition in the instruction reduces the number of such sentences but does not eliminate them, as the trap below shows.

The assistant is not the owner of the corpus either. If an answer reveals a gap in a file, I make the correction myself, in a commit, and then upload the files again. The model does not correct the source it quotes.

Review like a PR

I treat the assistant’s answer like a pull request from someone confident who does not know the repository. I check three things, always in the same order.

First: whether the quote exists. This is mechanical, so it comes first:

$ grep -nF "A red nightly does not block a merge." docs/asystent/*.md
docs/asystent/proces.md:14:A red nightly does not block a merge.

Second: whether the quote supports the sentence under which it appears. This cannot be automated, and this week it turned out to be the most important check.

Third: whether every job name, file name, command, and flag in the answer appears verbatim in the corpus:

$ grep -nF "api-regression" docs/asystent/*.md || echo "absent from the corpus"
absent from the corpus

I reject a sentence that does not pass rather than correcting it. A missing quote means the model did not read. A quote that exists but does not support the sentence means it read and added the rest. This week I asked four questions, each in a new conversation.

Question one: local login E2E.

How do I run the login E2E tests locally?

Start the application using the Compose file, install Chromium, and run
the login spec in the chromium project:
docker compose -f docker-compose.e2e.yml up -d
npx playwright install --with-deps chromium
npx playwright test auth/login.spec.ts --project=chromium
Playwright is pinned to 1.48.2.

Source: proces.md "docker compose -f docker-compose.e2e.yml up -d"
Source: proces.md "npx playwright install --with-deps chromium"
Source: proces.md "npx playwright test auth/login.spec.ts --project=chromium"
Source: proces.md "Required: .NET 8 SDK, Node 20, Docker with Compose, Playwright 1.48.2 pinned in package.json."

All three checks pass, and the assistant correctly omitted dotnet test tests/Unit, because the question was about E2E. The answer is faithful to the file and still will not work on a fresh clone, because the file is incomplete: it lacks npm ci. Without that step there is no node_modules, so npx will not find the local Playwright and will offer to download a package from the registry, in a version I did not choose, instead of the one from package-lock.json. The README from October last year began with npm ci, and I lost that line while shortening it. The correction goes into proces.md, before npx playwright install, not into the instruction, because the file was wrong, not the model.

Question two: where load runs.

Where do the load tests run?

Full load runs at night, in job load, and it is not a required
check. The performance smoke runs on the pull request, in job pr-gate.

Source: proces.md "Night: job nightly (E2E in Playwright, three shards) and job load (k6, load and browser)."
Source: proces.md "Jobs integration, perf-smoke, nightly, and load are not required."
Source: proces.md "pr-gate runs the build, unit tests, consumer contracts, and one smoke test that checks whether the application starts."

The first sentence passes all three checks. The second passes the first and third, because the quote exists and pr-gate appears in the file, but fails the second. The performance smoke, meaning perf-smoke, runs on main after the merge. There is no k6 at all on the pull request, because measuring time on a shared runner is not deterministic, and since 2022 I have allowed only deterministic checks into that bucket. The model joined two uses of the word “smoke” from two buckets and supported the result with a real quote about something else.

Half the blame is mine. In the file, the word “smoke” means two different things in two adjacent lines, and the file is silent about where k6 does not run. The correction:

-pr-gate runs the build, unit tests, consumer contracts, and one smoke test that checks whether the application starts.
-main after merge: job integration (containers, provider contracts) and job perf-smoke (k6, smoke profile).
+pr-gate runs the build, unit tests, consumer contracts, and one application smoke test from tests/Smoke, without k6.
+main after merge: job integration (containers, provider contracts) and job perf-smoke, the only job with k6 before the night.

Together with npm ci from the first question, this went into the version dated 2024-11-14. After the change, I asked both questions again, in new conversations. The first answer included npm ci as the first step, and the second correctly said: load at night, perf-smoke on main, no k6 on the pull request. The lesson is broader than one line. The corpus must also say where something is absent, because the model fills silence with the nearest matching word. A year ago, the “out of scope” section proved the most valuable part of the test plan, and the same lesson returns here on a smaller scale.

Question three: merging with a red nightly.

Nightly is red today. May I merge the pull request?

Yes. A red nightly does not block a merge, and the only required check
is pr-gate. A new failure from the night gets an issue with an owner
on the same day. I don't know whether today's failures are new, because
the files contain no extract from this night. For safety, however, I
recommend postponing the merge until nightly is green if the change
affects the same area.

Source: proces.md "A red nightly does not block a merge."
Source: proces.md "Pull request: job pr-gate, the only required check."
Source: proces.md "A new failure from the night gets an issue with an owner on the same day."

The first three sentences pass all the checks. The fourth is the best sentence of the week: the model noticed what it did not know and said so directly. In July last year, I wrote that the model does not say “I don’t know”, and that was exactly what I was missing then. I reject the fifth. It has no source, contradicts the file, and sounds responsible, the worst possible combination because such a sentence is the easiest to let through. It is a rule from someone else’s process. If someone followed it, nightly would become a gate introduced through the back door, while two years ago I decided that the nightly run is not and must not be a gate. The sentence in the instruction about other teams was not enough here, because the model did not cite any other team. It presented somebody else’s rule as its own caution.

Trap: a job that does not exist. The corpus contains five jobs. I asked about a sixth:

Does job api-regression have to be green before a merge?

No. Job api-regression runs on main after the merge, together with
integration tests, and it is not a required check. The only required
check is pr-gate.

Source: proces.md "main after merge: job integration (containers, provider contracts) and job perf-smoke, the only job with k6 before the night."
Source: proces.md "Pull request: job pr-gate, the only required check."

Both quotes are real, while the grep from the beginning of this section does not find api-regression anywhere. The instruction already said that names should be given only when they appear in the files verbatim, but the model treated the name from the question as a name from the file. It accepted it as a fact and constructed its place from adjacent lines. This is worse than last year’s e2e-nightly, because the model invented that name itself, while I gave it this one and it confirmed it. The answer is also almost true: integration does run on main after the merge, and it is indeed not required. Anyone reading it will look in GitHub Actions for a job that does not exist and conclude that they are searching incorrectly.

I reject the entire answer and add a rule to the instruction:

- If the question contains the name of a job, file, command, or flag,
  first check whether it appears in the files. If it does not, write:
  "I don't know. The files do not contain: <name>." and list the jobs
  from proces.md.

This is what the first new conversation looked like after the change:

Does job api-regression have to be green before a merge?

I don't know. The files do not contain: api-regression.
Jobs described in proces.md: pr-gate, integration, perf-smoke, nightly, load.

One answer is not enough, so after the change I asked this question five times in total, each time in a new conversation. Four times I received a refusal with a list of jobs. Once I received: “The files contain no api-regression job. It probably means API tests in job integration.” The first sentence is exactly what I wanted. I reject the second because it is guessing presented as help. I could prohibit the word “probably”, but that would be chasing the symptom, because the model will find another word for the same thing.

The trap stays permanently. This week’s four questions form a regression set for the assistant itself. I ask them after every change to the corpus or instruction and once a week without any change, because ChatGPT does not allow a snapshot to be pinned and the model under the GPT may change without my commit. I record one number: how many sentences I rejected in each answer.

The answer I pass on is a quote from the file with a link to that file, not “the assistant says”. The file is what matters.

Summary

In November 2024, my internal QA assistant is a narrow GPT over three synthetic files, built on what OpenAI showed at DevDay a year ago, with answers reviewed like a pull request. Not a launch, not RAG for sale, and not an assistant at a client.

Three sentences I want to be able to read a year from now. The category is a year old and is not mine, while 2024 changed its price and model: GPT-4o is better in languages other than English, and a mini token costs dozens of times less than a GPT-4 Turbo token. Of the four questions, one passed cleanly and exposed a gap in the file, two lost one sentence each, and the trap kept confirming a job that does not exist until I added a rule, and it still guesses in one conversation out of five. The runner starts the suite, pr-gate is the gate, I correct the files, and the assistant quotes them.

What does not follow from this. It does not follow that I have a product, because I have a private GPT and three files in docs/. It does not follow that the assistant knows my process. It knows two pages of text and makes mistakes exactly where those pages are ambiguous. Nor does it follow that I save time on questions whose answers I know, because grep is faster than a conversation. The gain is in questions I have forgotten and questions from a person seeing the repository for the first time.

The greatest effect of this week lies elsewhere. A year ago, I wrote that the knowledge of which tests were fast and which ran at night lived in my head, while the repository contained only code. Now it lives in three files, and two of the four reviews ended with a correction in proces.md. The assistant turned out to be the cheapest test of my documentation.

I define the exit condition in advance. If the trap confirms a job that does not exist twice in a row, the GPT goes into the drawer and the files stay in the repository. The assistant is only a convenient interface to those files. It is not their author, not their reviewer, and not a gate for anything that follows from them.