Why a recap
It is 15 December, and I am closing a year in which artificial intelligence stopped being a conference topic for me and became a code review topic. A year ago, I ended December by saying that ChatGPT was fifteen days old, none of my tests from 2022 had been created in a conversation with a model, and if I had anything to say about it beyond a first impression, I would write a separate post the following year. I wrote eleven. This post settles that promise rather than becoming a twelfth tutorial, so you will not find a single new configuration file here.
I am framing the question of the year in exactly these terms: did AI enter my QA team, or only the hype? This is not a question about sentiment because, for me, “in the team” means four very specific things. The tool has a named task. It has a place in the process, not in private experimentation after hours. It has a row in the pipeline or a deliberate reason not to have one. And it has a human who answers to the rest of the team for its output.
After twelve months, the answer is narrower than the headlines and broader than last year’s. AI entered the draft and the editor. It did not enter ownership. A model writes the first version of a test case list, the first version of a JSON fixture file, the first version of a test plan, and the first hypothesis for a flake. It does not approve a pull request, decide the regression scope, own a single ticket, or have a separate job in my CI. The sentence with which I will close the entire year is a copy of my June 2022 sentence about ghost text: the pace of drafting changed, but the boundary of responsibility did not.
There is also one detail that gets lost in writing about 2023 but matters today. The year’s loudest editor-side tool, GitHub Copilot Chat, is still not generally available on the date of this post. More on that in the IDE section.
Two years of foundations
I will start with what a year of language models did not erase because it is still a larger part of the answer than everything I added.
The axis from 2021 stands without correction. The 2021 recap was entirely about isolation: a container instead of an installation, a stub instead of someone else’s environment, a seed instead of random data, parallelism only when state does not leak. Twelve months with chat at hand did not give me a single reason to soften any of those statements. Quite the opposite: a model that produces test data in bulk tests whether my suite really has an owner for its state because a poorly isolated suite flooded with fresh synthetic data flakes faster, not less.
The axis from 2022 also remains, and it defined what this year looked like. Copilot left technical preview on 21 June 2022 with pricing of $10 per month or $100 per year, and throughout 2023 it remained the same tool for me: gray text in front of the cursor. In April 2022, I decided to move from Cypress to Playwright, and that migration progressed in 2023 exactly as migrations do: new browser tests are created in the new runner, while the old suite is still green and I am still not deleting it. July 2022 brought consumer-driven contracts, September brought the division of tests by trigger, and November brought a comparison of visual regression tools. All of these axes survived the year without a single change forced by AI.
This is the context in which the twelve months of new developments need to be read. I was not starting with an empty repository. I was adding a fast colleague to a suite that already had rules.
Chat and models
For me, the year in models looked like a staircase, not a leap.
In January, I described ChatGPT as a sketchpad for test cases. It is worth remembering what that meant in January: a six-week-old web product launched on 30 November 2022, with no paid plan, no API, and no GPT-4. I pasted acceptance criteria, received a list, removed half of it, and sent what remained to a ticket, not a commit.
In February, I published a post about the structure of a prompt for generating test cases: role, constraints, output format, and iteration instead of one shot. This is also the month when the experiment acquired a price because ChatGPT Plus launched in the United States on 1 February at $20 per month and became available outside the country on 10 February. That same month, on 7 February, Microsoft showed chat in Bing. I recorded it as a second chat on the market, not as a second tool in my workshop.
The gpt-3.5-turbo API appeared on 1 March, and this was the first date that genuinely changed my work that year because browser chat stopped being the only route to a model. Two weeks later, on 14 March, GPT-4 came out, so I wrote the post from 15 March one day after its release and had to make a distinction in it that people confused most often over the next four months: the model was available as a switch in the ChatGPT Plus interface, while the gpt-4 API was waitlisted. General availability of that API for paying customers did not arrive until 6 July.
In June, I closed the data thread with synthetic fixtures from gpt-3.5-turbo: first the data contract, then the prompt, then schema validation, and only then a file in the repository. Two days before that post, on 13 June, OpenAI added function calling, which I recorded as a fresh development rather than the core of the process. The contrast I have drawn around this subject for two years remains unchanged: the 2021 experiment with Completions and the davinci model reached the same conclusion as June 2023. Two calls with the same input do not have to return the same result, so this is a tool for inventing data, not a source of truth in a CI run. The endpoint and price changed. The nature of the artifact did not.
I spent July on triage. I pasted the test name, part of the log, and whether it had failed on a pull request or overnight, and I received three hypotheses to check: a race on a port, a shared fixture, or a retry masking a data problem. I verified each one by reproducing it, and not all of them held up. The model blamed “flaky Selenium” in a project that did not contain Selenium and proposed sleep as a solution. In one respect, this was my clearest month of the year: AI generates hypotheses faster than I do, but I remain the owner of the ticket because I am the one who can reproduce the problem.
There was competition this year, and I list it as headlines rather than tools I worked with: Anthropic’s Claude on 14 March, Bard on 21 March, Claude 2 on 11 July, and Llama 2 on 18 July. I did not build a single pipeline step on any of them.
Chat in the IDE
The second thread concerns the editor, and it has the year’s most confusing chronology, so I will set it out by date.
In April, I wrote about reviewing Copilot suggestions in C# tests. That post is about ghost text, not chat, because chat was not yet in my editor in April. Copilot X was announced on 22 March, with Copilot Chat as a technical preview behind a waitlist. I described the announced Copilot for PRs as it existed at the time, which meant another waitlist.
In May, I was admitted to the preview, and that produced a setup worth remembering because it shows the product’s stage clearly: VS Code Insiders plus the GitHub Copilot Nightly extension plus an active subscription. It was not a stable editor or a single click in the marketplace.
Availability then grew in stages beyond my control. On 20 July, chat entered public beta for organizations on a Copilot for Business plan. On 20 September, it entered public beta for individuals, five days after my September post about the pyramid, in which I still had to write that not every subscriber had access. On 8 November, at GitHub Universe, GitHub announced that general availability would come in December. Here is the point of this section, which I will state directly because it will sound implausible in a year: I am writing these words on 15 December, and Copilot Chat is still not GA. I have a promise from November and a beta in the editor. I do not have a generally available product.
That is why the AI bill for my workshop is short this year and fits into two official price list items: $20 per month for ChatGPT Plus and $10 per month or $100 per year for Copilot. Everything else is preview, beta, and waitlists.
One sentence about something people have asked me about since spring: Anysphere’s editor called Cursor exists, surfaced on Hacker News in March, and raised a seed round in October, but it was not my daily tool in 2023 and this recap has nothing more to say about it.
Playwright and the pyramid
The third thread is quieter, and it answers whether AI changed the shape of my test suite.
In August, I started working on component testing in Playwright. The package is called @playwright/experimental-ct-react, pinned at 1.37.0 in August, and the first word in that name matters more than the entire post. The preview dates back to 1.22.0 in May 2022, so it was not an August development, and after a year and a half it was still not GA. Mounting a single component in a real browser is faster than a full E2E test, and it works, but I do not connect an experiment to the file on which merging to main depends or build a project convention on it. I wrote the same sentence in December 2022 and did not have to correct it this year.
In September, I asked directly whether six months of chat and a chat preview in the IDE had changed the proportions of my suite. They had not. The test pyramid is not an invention from 2023. It comes from Cohn’s book and Fowler’s writing, and AI neither discovered nor flattened it. The bottom of the pyramid became faster because a helper and another [InlineData] are created more quickly. There is no shortcut through the middle because the model does not stand behind the contract schema or know more about my database than SQL does. The top remains as expensive as it was, so I still pay for E2E in wall-clock time overnight.
There is one practical consequence, and I have held to it all year: there is no “LLM” job in my CI. The trigger-based division from 2022 did not gain a fifth bucket for models. A pull request takes what is fast and deterministic, main takes real dependencies, and the overnight run takes full E2E and the engine matrix. If a model helped write a test, that test goes into one of these three buckets according to what it checks, not according to who drafted it. November’s visual regression matrix remains unchanged, with screenshots in the overnight bucket, and its hardest question is still who approves a new baseline, not which library compares the pixels.
The Playwright pin on which I am closing the year is 1.40.1 from 28 November.
Documentation and security
The final two months went to subjects that have less to do with AI than their titles suggest.
October was about test documentation written by a model. The conclusions are exactly the same as in January, only for a different artifact: an LLM is good for the first draft of a test plan, an acceptance criteria checklist, and a README file with instructions for running the suite, provided that its input is a ticket or code rather than the model’s memory of a typical application. Custom instructions, available since 20 July, were more useful here than a new model because they enforce language, format, and a ban on inventing jobs that do not exist. Review removes the same things every time: environments we do not have, docker compose confused with version one, and an invented Azure target. I treat merging documentation like merging code.
November covered security testing fundamentals for QA, and this is a gap a model does not close. OWASP Top 10 in its 2021 edition serves as a risk map instead of a checklist of four hundred tools, ZAP baseline serves as a smoke test in the test environment with a threshold for high-severity findings, and Burp is used manually for authentication and a synthetic resource without an ownership check. This is not a penetration test, a bug bounty program, or AI red teaming. It is the minimum at which green E2E tests stop being an argument that there is no hole in access control.
The November headline that cannot be omitted from a recap is dated 6 November. OpenAI DevDay introduced GPT-4 Turbo, GPTs, and the Assistants API. I record it as a date, not a process, because my November produced neither a single GPT nor a single line calling Assistants. The second headline is dated 14 November: .NET 8 was released. This year’s stack runs on .NET 7, and that is how I am closing December, for the same reason I did not adopt version seven in December last year.
What I am deliberately leaving for 2024
I am listing these items because in a year’s time I will not remember that they were decisions rather than oversights.
Autonomous tools and tests that repair themselves. Projects of this kind passed through the spring and half the conversations I had about testing this year, but they did not enter my suite for even a minute. I have no tool that corrects a locator by itself and pushes a commit by itself, and I do not want one until I can answer who signs off on that diff. This is a subject for next year, with an honestly measured result.
GPTs and the Assistants API. Eight days between DevDay and my November slot is too little time to write anything beyond a keynote summary.
Testcontainers Cloud. It has been in public beta since 25 January, meaning since the start of this year, and over twelve months I did not find a day to examine it properly. I am recording it as a subject, not an opinion.
NUnit 4.0.0 came out on 26 November. Three weeks before the end of the year is the worst possible time to rewrite a suite or publish the year’s tutorial, so I have read the change lists and saved them for January, exactly as I did with .NET 8.
Component testing remains an experiment because that is how the vendor labels it, not because I cannot be bothered.
Metrics
A year ago, I recorded two numbers for myself, and they are the only ones that interest me in this accounting.
The first is the wall-clock time of the required checks themselves on a pull request. After the trigger division in September 2022, my gate fell from thirty-four minutes to seven, and that number did not worsen this year, even though a new runner, new data, and plenty of code whose first version was drafted by a model passed through the suite. The second is the percentage of overnight runs in which something failed without a code change. This one is more important and harder because, in a year when generating two hundred new cases in fifteen minutes is easy, flakiness grows more quietly than run time.
The number I do not measure also has a name, and it is precisely this year’s temptation. I am not interested in how many models, plugins, and chats I have enabled. I am not interested in how many test cases a model wrote because the cost lies not in writing but in maintenance and reading the result. My drafting pace genuinely increased in 2023, and that is not the same as an increase in quality.
One sentence for 2024: I want my answer to “where is AI in this process?” to remain expressible in a single sentence, and I want that sentence still to contain the verb “review.”
See you in January.