Why a recap
It is 15 December, and I have just been through a year in which something appeared in my editor that had not been there before. I could turn this into a twelfth tutorial, paste one more configuration file, and leave it at that. I will not, for exactly the same reason as last year: anyone can reconstruct the list of tools that appeared on this blog since January from the archive in two minutes, and it gives nobody anything.
A recap makes sense to me only when it asks a question that a single post cannot ask because it sits too close to its tool. This year, I have two such questions, and they connect with each other.
The first: did artificial intelligence enter my everyday test writing, and if so, where exactly did it enter? Not in terms of headlines, but in terms of the keyboard. The answer I reached after twelve months is narrower and more boring than what I read online: it entered the editor as a suggestion for the next line, and it did not enter anywhere else.
The second: does the isolation I spent all of 2021 driving into the test suite still hold when I add a new browser runner to it? Because this is precisely the moment when last year’s conclusions face an exam. It is easy to write about clean state when nothing is changing.
In last year’s recap, I promised myself one sentence: that in 2022 I would measure the flake rate and the suite’s run time, not the number of tools I knew how to set up. I recorded it as an intention, and I will return to it at the end because this year that intention met a tool that tempts me with exactly the opposite metric.
Last year’s foundation
I will start with what this year did not erase because that is a larger part of the answer than anything I added.
I am not changing the axis from 2021. A container instead of an installation, a stub instead of someone else’s environment, a seed instead of random data, parallelism only when state no longer leaks. Testcontainers with MSSQL and retry policies with Polly remain my canon, with the same caveat still applying: retry cures a temporary failure of someone else’s service, not a lack of isolation. This year gave me no reason at all to soften that sentence.
I began the year with a post that, from the outside, looks the least fashionable of the entire twelve months, but for me it is the foundation of everything that came later. In January, I laid out test organization in xUnit and NUnit on .NET 6: xUnit 2.4.1 with a new class instance for each test and explicit sharing through fixtures and collections, NUnit 3.13.2 with FixtureLifeCycle as the first step in an old suite. The conclusion of that post was: fixture organization is a prerequisite for parallelism, not its consequence. After a year, I am adding a second sentence. Fixture organization is also a prerequisite for adding new layers sensibly because a suite in which I do not know who owns the state will not accept any new tool without a fight.
Two pieces of infrastructure changed their names this year, not their strategy, and I want to record them so that nobody reads last year’s posts as a list of mistakes. Compose version two has been generally available since 26 April, and on new jobs I call it without the hyphen. My posts about Docker for QA and about a test environment with Compose teach the hyphenated form, and they will stay that way because this is a command change, not a change of approach. Similarly, Selenium Grid in Docker remains a tool for a browser matrix, not for a single scenario on every push.
Copilot: preview, boilerplate, GA
The loudest thread of the year consists of three posts and one flat line between them.
In February, I described the GitHub Copilot technical preview. It is worth remembering what that meant in February: a waitlist, free access for those admitted, VS Code in the 1.64 line, and gray text in front of the cursor. The preview announcement was on 29 June 2021, so even then I was not writing about a launch, but about a tool that had existed behind a gate for over half a year.
In March, I published a more specific post: Copilot with NUnit and xUnit test scaffolds. That was the point when I stopped describing the extension and started describing my own work. I repeat that month’s result without changes because nine months have not shifted it: about one third less typing and zero percent of the decisions. Another [InlineData], a constructor assigning fields, a block of using directives at the top of the file, and the skeleton of a method whose signature I wrote myself - yes. An assertion, a locator, choosing data for a branch, and a fixture lifecycle - no, and those four areas were systematically probable rather than correct.
On 21 June, Copilot left technical preview. The waitlist disappeared and pricing appeared: $10 per month or $100 per year for individuals, a 60-day free trial, free access for verified students and maintainers of popular open source projects, and free access until 22 August for people from the preview. I wrote then what is, for me, the most important sentence about this tool all year: the access gate and billing changed, but the way of working did not. Still Tab and Esc, still gray text, still nowhere I could ask anything.
The rest of the product changes this year concern availability, not category. The Visual Studio 2022 extension was released on 29 March, so since spring there has been no need to stay in VS Code. Since June, the settings have included an optional filter that blocks suggestions matching public code in blocks of roughly one hundred and fifty characters or longer. The filter is a switch, not a tool for checking the provenance of code, and it does not remove a single minute of reading the diff from my work.
The contrast I have to draw for the fourth time this year because it returns in every conversation. In June 2021, I described an experiment with generating test data through the Completions API: a POST to an endpoint with davinci in the address, a hand-written few-shot prompt, temperature and max_tokens to set, text in the response, schema validation, and only then a file in the repository. The result was data, meaning an artifact that someone reviews in a pull request. Copilot is on the other side: the prompt is my open file, there are no parameters, and the result is not an artifact, but the way my code was created. After I press Tab, it cannot be distinguished from what I typed with my own fingers, and that is exactly why review before the commit is more important with this tool, not less.
To sum up the year in one sentence: artificial intelligence entered my editor as a prediction of the next line and stayed there. I pay for it as a subscription from my personal account, enable it only where I have clear permission from the code owner, and have developed the habit of rejecting four categories of suggestions without reading them to the end.
Playwright: decision, setup, pipeline
The second thread of the year is quieter and cost me more work.
In April, I wrote why I was starting to move away from Cypress toward Playwright. I began that post with a sentence I will repeat here because it is easy to lose in a year-end recap: Playwright is not new. Release 1.0.0 is from 6 May 2020, and both the official @playwright/test runner and Trace Viewer are from June 2021. Only my decision was new. I listed four reasons: three engines from one API, including WebKit on Linux, a new BrowserContext for every test, a trace from the CI agent instead of video, and a runner with workers in the package instead of a service with a key. The pin in that post is 1.21.0 from 12 April.
In May, I set up an empty project on 1.22.0, and two conclusions from it survived the entire year. I pin the version from the first commit because the package brings specific browser builds with it, and a caret in package.json means silently replacing an engine in the middle of a sprint. The weight of configuration moves from the test to one file: parallelism, retries, artifacts, and the browser matrix are four fields in the config, while the test stays short.
In August, both of my layers went into one pipeline file: the Testcontainers NuGet package in 2.1.0 on the integration job and Playwright 1.25.0 on the browser job. Incidentally, the .NET package changed its name from DotNet.Testcontainers in June, and that is the only thing from that post that confuses people more than the YAML itself. Three conclusions from August remain: two jobs are easier to read than one long job, a container started in one job does not exist in another, and the Testcontainers job runs on Linux because the Docker daemon on a hosted Windows agent will not start a Linux database image.
Now the part I will not write, although it would be more pleasant. I do not have a rewritten suite, and I will not give a percentage. I have a decision from April that new browser tests are written in Playwright, I have a handful of smoke tests on CI, and I have questions to which I do not know the answers, such as how many of the existing Cypress tests are worth migrating at all instead of rewriting or deleting. This is the start of the migration, not its end.
And a sentence that must be stated directly in a recap because a twelve-month arc can easily be read as a verdict. Cypress is not dead. The suite I run is still running, the series of posts from refactoring App Actions versus the Page Object Model through the pipeline on Azure DevOps to publishing results remains on the blog, and Cypress released two major versions this year: ten in June and eleven in November. I am changing the tool for new work, not announcing the funeral of what works.
API: contracts and in-memory
The third thread contains neither a browser nor a language model, and in my opinion it delivered the most this year.
In July, I added consumer-driven contracts with PactNet to my stubs, using version 4.1.0 on .NET 6. The reason the topic returned is technical, and I want to name it precisely: the idea itself is from 2006, Pact from 2013, and what is new is the 4.x line for .NET, meaning one NuGet package with a Rust core and an in-process mock server instead of per-system packages. A contract closes a gap that neither WireMock stubs nor a container with a real database closed: a stub is my idea of someone else’s response, and it stays silent on the day the other side changes a field. A contract is a feedback channel into someone else’s pipeline.
In October, the post about DI for testing in .NET with WebApplicationFactory was published, and it is a third path to the same application that is worth comparing with the previous two instead of choosing one of them. The host lives in the test process’s memory, I replace dependencies in the DI container, there is no Docker, no cold image pull, and no browser. The price is obvious: this is not a real database server, and it is not a real rendering engine.
By the end of the year, this gave me a fairly simple accounting of costs. I use the in-memory host for API-layer logic and wiring dependencies together. I use a container when I check something about which SQL knows more than my idea of SQL. I use a contract at the boundary between my system and someone else’s. I use a browser for what a person actually sees, and I use it least because it is the most expensive. None of these four things replaces the others, and the biggest mistake I see in my own archive is trying to substitute one for another because it was at hand.
Trigger strategy
September’s post about test strategy in CI/CD was the point when I stopped adding tools and started arranging what I already had.
The division came down to one criterion: what happens if this test fails ten minutes after the merge instead of ten minutes before? A pull request takes what is fast and deterministic: unit tests, consumer-side contracts, and one smoke test with its own state. main takes real dependencies, meaning integration with a database in a container and provider-side contract verification. Nightly takes full E2E and the engine matrix, with fail-fast disabled because I want the complete picture from a nightly run, not the first failure.
The number this gives me is the only number in this recap, and I deliberately do not turn it into a success: my pull request gate dropped from thirty-four minutes to seven, while the total machine time did not decrease by a single minute. The same tests still run, only elsewhere. I did not make anything faster; I moved the waiting from a person to a schedule.
The prerequisite for this entire arrangement is the order I set out in parallel tests on CI a year ago and did not have to correct this year. Isolation first, then wall-clock. Dividing a poorly isolated suite by trigger gives exactly the same result as running a poorly isolated suite in parallel, meaning faster flakes. This is joined by the mechanics from Testcontainers in Azure Pipelines: Docker sits on the agent host, not inside a container job, and that has not changed over the year.
One temptation returns every quarter, so I will record it here too. Retrying is not a pass from nightly to pull request. A test that needs three attempts to fit into the gate does not belong in the gate. This applies equally to retries in the Playwright configuration and [Retry] in the .NET suite.
Visual on 15 November
I spent November on comparing visual regression tools, and this is a good place to show what the year changed in this area without writing a second tutorial.
There is one axis: where the baseline lives and who approves its change. A baseline in the repository is provided by Playwright in its core because the screenshot assertion has been in the runner since 1.22.0 from May, and by BackstopJS 6.1.4 with its own config and the Playwright engine since the 6.0 line. Percy gives me a baseline in the cloud and review in a web interface, while Applitools adds its own layout comparison layer as the Eyes product. Separate from these is the path I once chose myself: a Cypress plugin from 2020, meaning cypress-image-snapshot in 4.0.1, from before the plugin model changed in version ten.
The year’s change fits in one sentence: comparing images no longer requires a separate tool for me because it entered the core of the runner in which I had started writing new tests anyway. Everything difficult stayed where it was, and none of it is a library question: tolerance thresholds, rendering differences between my laptop and the CI agent, and the name of the person who approves a new baseline. That is why, in line with September’s division, screenshots sit in my nightly bucket, not at the gate.
The post from 15 November was published two weeks before the loudest announcement in this industry this year, and it does not say a word about it because that product did not exist then. I am leaving it in the archive as it is.
Year-end headlines
Three things from the last six weeks exist and will appear in writing about 2022, so I am recording them as headlines, not as tools I worked with.
On 30 November, OpenAI announced ChatGPT. That was fifteen days ago. I have an account there, I clicked it like everyone else, and I will leave it at that in this post because a year-end recap should describe the year. None of my tests from 2022 was created in a conversation with a model, no fixture in my repository came from chat, and no step in my pipeline calls any language model. Treating this announcement as the theme of the entire year would move its date back by eleven months. When I have something to say beyond a first impression, I will write a separate post, and it will be a post from next year, with an honestly measured result, not a paragraph added to a December recap.
GitHub Copilot for Business appeared on 7 December. It is the same product line I described in June, only with a plan for organizations: on GA day, only individual subscriptions could be purchased, while the business version was announced for “later this year.” Now it is an item in the price list. For me, this changes one question, and it is not a technical one: with an organization plan, the conversation about which repositories it may be enabled in stops being my private decision and becomes the code owner’s decision. The habit of reading the diff before a commit does not change by a millimeter.
.NET 7 was released on 8 November. My entire pack of posts from this year stands on .NET 6, and that is how I close the year because December is a bad month to replace the foundation under a regression suite. I am recording the date so that in a year’s time it will be clear that the version was available and that I consciously did not adopt it.
What I deliberately leave out
Four things were within reach and did not enter my suite. I am listing them because in a year’s time I will not remember that these were decisions, not oversights.
Component testing in Playwright has been available since May, together with the screenshot assertion, and is still marked as experimental. I do not connect an experiment to a file on which merging to main depends, and I do not build a project convention on it.
Library versions remain where I pinned them. Testcontainers in the 2.x line, NUnit in 3.13, xUnit in 2.4, Playwright on 1.27.1 from October, even though 1.28.0 has been available since 16 November. Raising pins in the middle of December is the worst possible time for this work, and I have read the change lists and saved them for January.
Generating test data through a conversation interface. I have already conducted the Completions experiment from 2021, and its conclusion still applies: two calls with the same input do not have to produce the same result, so this is a tool for the data invention stage, not a source of truth in a CI run. Fifteen days with the new interface do not invalidate that conclusion, and I am not going to add a layer to the test data management strategy on that basis.
Rewriting what works. The Cypress tests remain, the page object refactor in C# remains the rule for Selenium code, and the posts from 2021 remain unchanged for new command names. A green, maintained suite is not technical debt merely because it was created with a different tool or in a different year.
One sentence for 2023
One sentence: next year, I want to bring the UI migration to a state where one suite covers what two cover today and continue measuring the same two numbers as this year.
The two numbers are the wall-clock time of the required checks themselves and the percentage of nightly runs in which something failed without a code change. The first says whether anyone is still waiting for the result. The second says whether looking at the report in the morning makes sense. Improving the first at the expense of the second is easy and always ends the same way.
I also want to name the number I will not measure because it is the temptation of this particular year. I am not interested in how many models and extensions I have enabled in the editor. My pace of writing code genuinely increased in 2022, and that is not the same as an improvement in quality; the point where it is decided whether a test suite means anything still lies in the process, not in the editor. A year ago, I closed the 2021 recap with a sentence about tools versus process, and after twelve months with a faster tool I see the same thing, only more sharply.
See you in January.