2024 in QA - new AI channels, the same ownership

Why a recap

It is Sunday, 15 December, and I am closing a year in which AI found several new addresses in my work and signed off on no merges. A year ago I wrote a sentence in the December recap that I had warned would sound unbelievable a year later: on the day of that post, Copilot Chat was still not generally available. That sentence was out of date fourteen days later, because general availability came on 29 December, and in another fourteen days it will be a year since chat took its place in the same stable VS Code as the rest of my work. I wrote eleven posts this year. This one is an account of them rather than a twelfth tutorial, so once again you will not find a new configuration file here.

In December 2023, I asked whether AI had joined my QA team or only the hype, and broke “on the team” down into four conditions: a named task, a place in the process, a row in the pipeline or a conscious reason for not having one, and a human accountable for the result. This year I am narrowing the question to the fourth condition: did the new AI channels change who signs the merge? By a channel, I mean any place where a model leaves text for a human to read. By ownership, I mean the signature under the statement that a change is ready for main.

After twelve months, this is the answer. In 2024, AI got new channels in my QA: Chat GA in stable VS Code, a comment on a pull request, a screenshot description, a nightly report draft, and an assistant instruction. It did not get merge ownership, and it did not cure CI flakes through “self-healing”. On 15 December, Copilot code review, a separate product for automatic pull request reviews, is still not generally available. The pyramid from 2021-2023 stays unchanged. In short: more places for a draft, the same human at the merge.

Three years of foundations

I will start as I did a year ago, with what the new channels did not erase, because that is still the larger part of the answer.

The 2021 axis still stands. The 2021 recap was about isolation: a container instead of an installation, a stub instead of someone else’s environment, a seed instead of random data. Testcontainers for .NET from April 2021 remains my runtime canon, and the cloud in July did not change that. The Completions experiment came from the same year, and its conclusion returned more often in 2024 than I expected: two calls with the same input do not have to produce the same result, so a model can draft, but it cannot be the step on which a merge depends. I returned to that conclusion in January with an agent loop, in June with a pull request bot, and in October with a nightly report draft, each time with a different model.

The 2022 axis stays too. The 2022 recap closed the year with Copilot in the editor and the start of Playwright. Copilot left technical preview on 21 June 2022, and its gray text survived all of 2024 without changing roles for me: chat did not replace it, but stood beside it. The split of tests by trigger from September 2022 still has three baskets: pull request, main after the merge, and nightly. This year I added a bot, cloud runtime for containers, axe, and k6, and everything went into the existing baskets rather than a new one. The comparison of visual regression tools from November 2022 remained the pixel gate throughout a year in which models learned to describe images.

The 2023 axis is the boundary I tested this year: AI in the draft and in the editor, not in ownership. In April 2023, I reviewed Copilot suggestions in C# tests, and of 41 accepted multiline suggestions, 12 remained unchanged. In May, I got Copilot Chat in preview in VS Code Insiders with the Nightly extension. In July, model-assisted flake triage produced 33 hypotheses, 9 of which I confirmed by reproducing the failure. In September, the pyramid did not flatten; only its bottom got faster. So 2024 began with chat generally available since 29 December 2023, not with new ownership.

Headlines that did not enter the process

The first quarter was a review of the promises of autonomy that were loudest in discussions about testing. Where I could, I tested them in the same way: on a synthetic demo, with numbers, and with the question of who signs the result.

In January, agents. The Auto-GPT repository was created on 16 March 2023, and its first tag is dated 12 April 2023, so the subject was loud rather than new. Over four evenings, I gave the loop the goal “fix the failing login test” and a directory of tests. Three of the four steps in the recorded run had to be crossed out: a data-testid that did not exist, a waitForTimeout(5000) that did not even help, and git commit -am without reproducing the failure and without a pull request. The real cause was outside the repository, in the consent modal flag that I had enabled myself on 8 January in the local application’s configuration. An agent that “fixed” this test would have erased the signal. The gain was narrow: in two runs out of six, I got a list of suspects sooner.

In February, self-healing. The hype concerned the mechanism, not a launch: Mabl has existed since 2017, Healenium has had a 3.x line since 9 October 2020, and Testim was already advertising Smart Locators in the announcement of its acquisition by Tricentis on 9 February 2022. I wrote thirty lines of heuristics without any model and ran two red cases through them twenty times each. The heuristic repaired a changed button id 20 times out of 20, and that was an honest result. It also turned an /auth endpoint that had stopped responding green 20 times out of 20, because it switched the assertion from the “Dashboard” heading to “Log in”. The report did not distinguish one green from the other. Environment flake is not cured by replacing a locator, Playwright through 1.49.1 has no official self-healing, and the 2023 triage stays in its place: reproduce first, form a hypothesis second.

In March, two headlines at once. On 12 March, Cognition announced Devin as an “AI software engineer”, and I recorded the date because I did not have access to it and had nothing to measure. Copilot Workspace was then an announcement from Universe on 8 November 2023. On 29 April, it entered technical preview with a waitlist, and on 15 December it is still not generally available. The condition I set in March was a preview that I could run myself, with the same issue producing the same set of files in two runs. By December, I had not shown a single such run here, so I do not write about Workspace as a tool.

Three months, three promises of autonomy, and one sentence that passed from January to December unchanged: the suite has no autonomous owner. It has isolation, a trace, and me.

The IDE and an E2E pair

The gray text from 2022 stayed in the editor under the same contract: it suggests a line, and I either accept it or not. This year’s addition is chat beside it, in the same stable editor and under the same 10-dollar subscription. In May 2023, I needed Insiders and the Nightly extension for that.

In March, I separated two things that tended to blend into one in conversation. @workspace is an agent in Copilot Chat, described in the changelog from 30 November 2023, that gathers context from several repository files and answers with text in chat. Copilot Workspace is a separate product intended to take a change from an issue, through a plan, to a pull request. When I asked about several test files, @workspace gave me three pointers. I accepted one and crossed out two: the first added an assertion for an incorrect password to the end of the successful login test, and the second imported from a ./locators module that did not exist. Copilot Enterprise from 27 February, with chat on github.com and pull request summaries, is not Workspace either, and I do not have it on my account either.

In April, I made an E2E pair from this: chat beside the spec, runner 1.43.1, and one acceptance criterion, the incorrect-password message. Five drafts, two pasted in, three crossed out, and each rejection was caught by someone different. The runner caught a getByTestId('login-error') that did not exist in the application, at the cost of one timeout. The compiler caught toHaveAccessibleName, which did not exist in 1.43.1, at the cost of one second. The irony of that rejection appeared later: the assertion arrived in 1.44.0 on 6 May, three weeks after the post, and in August I made it the core of the a11y automation. Only I caught the “Welcome back” heading that was always on the page, because the assertion was green from its first run. The pair has a budget of two returns to chat; on the third, I write the assertion myself, and I am the one who reads the trace from the retain-on-first-failure mode, which arrived in 1.43.

General availability of chat therefore changed the address of the draft, not the address of the verdict. The draft is now created beside the spec, the runner issues the verdict, and the signature stays with me.

Eyes and gates

In May, I checked what vision models said about screenshot pairs that the gate had already judged. The gate was toHaveScreenshot on 1.44.0, while gpt-4-turbo with vision and Claude 3 described the screenshots for me over two evenings. With 18,420 different pixels, the gate failed the build, and the model wrote three sentences, one of which claimed that the layout had not changed. A font shifted by two pixels produced 3,112 different pixels, or 0.0034 of the whole at a threshold of 0.01, so the gate stayed silent and the model wrote that the images looked the same. In the gate, I had a number I could change, and I changed it to maxDiffPixels: 400. In the model, there was nothing to change. On the third pair, the model described a “Start now” button that was absent from both screenshots. GPT-4o was two days old on the day of the post and got one sentence. The model’s description remained my private magnifying glass, while the pixel gate remains what I chose in November 2022.

In August, I added inexpensive a11y automation: AxeBuilder from @axe-core/playwright 4.9.1 and toHaveAccessibleName plus toHaveRole on Playwright 1.46.0. On the broken form, axe reported button-name and label, both critical. The more interesting cases were two refactors on which axe stayed silent: a button with a copied aria-label="Close" and a div in place of a button. The assertions caught them because I had written into them what the control was supposed to be called and what it was supposed to be. The rules under my tags touch 20 of the 55 WCAG 2.2 success criteria at levels A and AA, and usually only partially, so a green job is not a WCAG certificate. Screenshots of the broken and repaired form were identical byte for byte, while a button shifted by 96 pixels was caught by the visual test, not axe. Visual catches appearance, the accessibility tree catches name and role, and neither replaces the other. The @smoke-tagged a11y test runs on pull requests, while the full scan, including dark mode, runs at night.

Three months later, the August post began to age. On 18 November, in 1.49.0, Playwright gained toMatchAriaSnapshot, an assertion that compares a page with an expected accessibility tree stored in YAML. It is absent from the August tutorial because it was absent from the runner. I name it here, but I am not rewriting the automation yet. In August, I wrote that a11y had no baseline to overwrite. An expected tree in YAML is a baseline, and every baseline brings the question I recorded a year ago as the hardest one in visual regression: who approves the new one?

The Playwright pin on which I am closing the year is 1.49.1 from 10 December. Component testing still has the word experimental in its package name, and the runner still has no official self-healing.

CI and runtime

In June, a model got a voice on a pull request. The Copilot Enterprise summary is outside my subscription, so I tested PR-Agent in GitHub Actions. The Codium-ai/pr-agent@v0.22 action is pinned to a tag that pins a one-line Dockerfile, not the bot’s code or the model, and that is the first thing to know about this job. On a synthetic pull request, the bot left four comments. I accepted one, about the shared TestUsers.Default account. I crossed out three: “Relevant tests: No” when the spec was in the diff, [Retry(3)] as a cure for instability, and toBeVisible() in place of toHaveText(). Every rejected comment would have left the pipeline green. The bot has an advisory voice in a separate, non-required job; it has neither a veto nor a signature, and the “Allow GitHub Actions to create and approve pull requests” option stays disabled. This is a different channel from the gray text of April 2023: that was visible to one person with an open buffer, while this is visible to the whole team.

Autofix for CodeQL moved from public beta on 20 March to general availability on 14 August. It is a security tool: it proposes a fix for a code scanning alert and does not judge whether an assertion checks anything, so I do not count it as a test reviewer. Copilot code review is still not generally available on 15 December, and I am not building anything on it in this recap.

In July, I moved the container runtime to Testcontainers Cloud, a Docker product following the acquisition of AtomicJar announced on 11 December 2023. The public beta dates to 25 January 2023. I do not give a general availability date because I did not find an announcement that dates it, and I will not invent one in a recap. July’s most important sentence concerns code that did not change: the same ContainerBuilder runs locally and in the cloud, the test and assertion are untouched, and preparing the synthetic set for the cloud cost one line, replacing WithBindMount with WithResourceMapping, because the worker cannot see my disk. The second sentence concerns a silent green. Version 3.9.0 of the library takes the address from tc.host before DOCKER_HOST, and when nobody answers at that address, it falls back to the local socket without warning and the test passes on a different runtime. That is why the agent step in CI is red when the agent does not connect. The Cloud agent runs only in the integration-test job, outside the pull request gate, while the local daemon from 2021 remains the canon.

In July, I also described the billing from the documentation current at the time, with a per-person desktop charge and worker minutes on CI, and noted that if Docker changed it, that paragraph would age, not the rest of the post. New Docker plans launched on 10 December, under which Testcontainers Cloud no longer has a per-person license and runtime minutes are included in paid subscriptions. That is a December pricing change, not a correction to July. The July paragraph describes July, and the code from that post has not changed.

Load, report, assistant

In September, I added k6 separately from E2E. On a synthetic login with a regression in which the password hash was computed synchronously with PBKDF2, Playwright 1.47.1 was green, and so was a k6 smoke test with two VUs. Only a ramp to fifty VUs revealed a p95 of 915.45 ms at a 300 ms threshold and exit code 99, across 2,471 requests, all of which received 200. k6/browser in 0.53.0 roughly resembles Playwright, but it has neither getBy* nor expect and knows only Chromium, so k6 does not replace the functional suite but stands beside it: protocol smoke on main, load at night, nothing on the pull request. The most expensive lesson of that week was green too, because a threshold on a metric without a single sample passes, which is why the browser script keeps a guard in finally. OpenAI o1-preview from 12 September remained a headline: I derive the traffic profile and thresholds from measurements, not from chat.

In October, I returned to the nightly run from the other direction. The 2023 triage was a microscope trained on one failure, while October’s draft is a map of the whole night. A 26-line script, without any model, turns three JUnit files from the nightly matrix into a 34-line extract and marks new failures by itself. Only then does gpt-4o-mini-2024-07-18, pinned to a snapshot, receive the extract and turn it into a paragraph for just under four hundredths of a cent. With twelve failures, including one new one, the model ordered them well and drew the wrong conclusions: I crossed out a fabricated nightly-api job, WebDriver with advice about implicit wait in a project without WebDriver, and a summary that blamed the night on the environment and told me to rerun it. A human publishes the paragraph, while the runner’s exit code sets the night’s color. Since October, I have recorded how many sentences I cross out in every draft, and the exit condition is this: if crossing things out starts to cost more than writing from scratch, the draft leaves the process.

In November, I assembled an internal QA assistant, and that word needs to be narrowed immediately. It is a private GPT with an instruction made from the same four blocks as in February 2023 and three synthetic files: proces.md, konwencje.md, and nightly.md. Capabilities are disabled, there are no Actions, it has no token for anything, and the files contain no client name. The prior art comes from DevDay on 6 November 2023, so this is a practice, not a product launch. Of four questions, one passed cleanly and, precisely because it did, revealed that npm ci was missing from proces.md. Two answers lost one sentence each, and one of those sentences was the rule “wait for a green nightly”, which did not exist in any file. The model kept confirming the trap involving a nonexistent api-regression job until I added a rule; after the change, it refused in four conversations out of five. Two of the four reviews ended with a correction to proces.md, so the assistant turned out to be the cheapest test of my documentation. The exit condition is written down in advance: if the trap confirms a nonexistent job twice in a row, the GPT goes on the shelf and the files stay in the repository.

I am leaving two autumn headlines as dates. .NET 9 came out on 12 November, while the projects in the examples repository close the year on .NET 8, the LTS release, because rewriting the set in the final weeks of the year is a bad idea, just as it was with version eight a year ago. On 22 October, Anthropic presented the refreshed Claude 3.5 Sonnet and computer use in public beta, meaning a model that looks at a screen and clicks. It did not click anything in my process.

What I deliberately leave for 2025

I am listing these for the same reason as a year ago: in twelve months, I will not remember that they were decisions rather than omissions.

First, an account of last year’s list. I left five items for 2024, and three of them got their own posts this year: agents and self-healing in January and February, Testcontainers Cloud in July, and GPTs and Assistants in November. Component testing still has the word experimental in its package name, and NUnit 4 did not get a separate post.

Copilot code review as a generally available product. June’s condition applies to every automatic reviewer, regardless of the logo: required status only when the code and model are pinned to a version, two runs on the same pull request produce the same comments, and not once does it propose a retry or a weaker assertion as a path to stability. Even then, a veto, not a signature.

An agent that maintains the suite by itself and merges by itself. The January condition has two halves: two runs on the same input produce the same diff, and the proposal knows how to say “I don’t know, ask a human about the environment”. November’s GPT came closest to the second half this year, even though it is not an agent: after I added the rule, it said “I don’t know” in four conversations out of five. No channel I tested this year met the first half.

Testcontainers Cloud with a general availability date, if I ever find an announcement that dates it. Until then, I write “Docker product” and do not invent a day.

toMatchAriaSnapshot in the a11y automation. It will enter when I can say who approves a change to the expected tree and how that approval differs from overwriting a screenshot baseline.

The Cursor editor exists, but I did not switch to it this year and I am not making it the tool of the year.

Metrics

Last year, I recorded two numbers: the wall-clock time of required checks on a pull request and the percentage of nightly runs in which something failed without a code change. Both stay. I am adding the question that proved more important than either in 2024: does the assertion check anything?

Wall-clock. Of what I added this year, one thing entered the required pull request checks: the a11y smoke from August, which costs a fraction of a second on the login form. The bot from June runs in a separate, non-required job, the k6 smoke and Cloud agent run after the merge on main, and load plus the full a11y scan run at night, while the screenshot descriptions, report draft, and assistant are not jobs at all. If the pull request gate ever slows down, it will not be because of AI, because there is no AI on it.

Flake. Here, the year taught me something I had not expected. Four different mechanisms promised a lower flake rate without fixing the cause: waitForTimeout(5000) from the agent loop in January, a locator replacement in February, [Retry(3)] from the bot in June, and in October advice about implicit wait plus a summary that told me to rerun the night. Each would have improved the number I measure and damaged what that number is meant to measure. None got a place in the pipeline. The flake rate is a good metric only as long as nobody optimizes it instead of the suite, and in 2024, for the first time, I had tools at hand that were eager to do exactly that.

Does the assertion check anything? The year’s most expensive errors were not red. February’s heuristic turned an /auth outage green. April’s draft asserted a heading that was always on the page. May’s two-pixel shift fit below the threshold. The bot’s three rejected comments in June would have left the pipeline green. July’s library would have silently returned to the local socket. Axe stayed silent at “Close” in August. Playwright was green in September while p95 exceeded the threshold by more than three times, and a threshold without samples always passed. A red test announces itself. A green test that checks nothing waits for someone to read it.

The third number arrived in October: how much I crossed out from what the model wrote. Collected from the year’s posts, it looks like this:

Not one row contains a zero. These are one person’s notes from synthetic demos, not a benchmark, and I am not comparing the rows with each other because each channel got a different task. More important than the numbers is the column the table does not have: who caught the error. In April, I divided it among the runner, the compiler, and a human. The runner and compiler caught what was red, while the human caught what was green, and after twelve months that is the shortest description of why review exists.

The number I do not measure has the same name as a year ago. In this year’s posts, I recorded seven model launches: Gemini 1.5 Pro, Claude 3, GPT-4o, Claude 3.5 Sonnet, GPT-4o mini, o1-preview, and the refreshed Claude 3.5 Sonnet. One has a pinned snapshot in my process, gpt-4o-mini-2024-07-18, and it sits in a draft that a human publishes. The number of models says nothing about the quality of the suite. The number of rejections at least says that someone was reading.

A year ago, I wanted the answer to “where is AI in this process?” to remain expressible in one sentence and for that sentence to contain the verb “I review”. At the end of 2024, it reads: the model drafts in five channels, while I review every draft and sign the merge myself. One sentence for 2025: I want that sentence still to contain the verb “I review” a year from now, even if there are ten channels.

See you in January.