The test pyramid and AI - September 2023

I am not asking about the tool of the year, but about proportions

It is 15 September. Since January, I have kept a browser tab with chat open next to my editor; since March, I have had a GPT-4 switch in my subscription; since July, access to the gpt-4 API without a waitlist; and since spring, chat in the editor itself. I have written four posts about this this year: about the first test cases from chat, about prompt structure, about GPT-4 the day after launch, and about reviewing Copilot suggestions. They all answered the same question: does this tool shorten a specific activity I perform at my desk?

Nine months is enough to ask a question one level higher, and it is the only question in this post. Not “which tool is best,” but this: after these nine months, have the proportions of my test suite changed? Do I have fewer E2E tests because the model helps me write tests? Has any new layer appeared that was not there a year ago?

I want to make one thing clear from the start so there is no misunderstanding about what kind of text this is. The test pyramid is neither a 2023 discovery nor my idea. Mike Cohn described it in 2009, Martin Fowler popularized it in a note in 2012, and over a decade it has acquired its own critics, trophies, and other shapes. I am not proposing a new framework here, naming anything after myself, or inventing an “LLM tests” tier. I am only checking whether the old diagram still describes what I actually have in the repository.

In my 2022 summary, I wrote that AI had entered my editor as a prediction of the next line and had not entered anywhere else. That sentence stopped being true sometime in February. It is worth examining exactly how it stopped being true, because the difference between “I write tests faster” and “I need different tests” is the whole point of this post.

I am collecting the numbers from the same public RealWorld-style demo that I have used on the blog since 2021: login, the logged-in user’s header, an article list, and several endpoints on the .NET side. There is not a single number from a client project here, and there will not be. My runner is pinned to 1.38.0 from 13 September, and the backend to .NET 7.

I counted files and test methods at two points: in December, when I was writing the year-end summary, and last weekend.

The E2E share fell from 8 percent to 5 percent. That sounds like a flattened pyramid, but it is not, because the denominator grew by 160 tests while the numerator did not grow at all. I did not remove a single E2E scenario because I have a model. Three that disappeared were removed because a feature was deleted from the demo, and one new scenario was added when I introduced a shopping cart. That is the entire change in proportions: the bottom grew faster than I could have made it grow a year ago, and that is all.

What AI actually does at the bottom of the pyramid

The bottom of the pyramid is the layer where AI earns the most for me, and it is also the layer that was already cheap. This combination is more important than it looks: I got the speedup where it did not hurt anyway.

I write unit tests and helper classes with Copilot ghost text, and that has not changed since last year. Another [InlineData], a builder overload, mapping a DTO to a model, an empty page object class with a list of locators to fill in. In those cases, I accept the suggestion with almost no changes because the shape is repetitive and does not depend on my application’s behavior. Most of the 117 new unit tests in the table above were created this way: I write the name and the first assertion, the editor completes the rest of the line, and I read it. The filter has not changed, and I described it in detail in April: the assertion, the data, the cleanup, and the selector that does not exist.

I generate test data with the gpt-3.5-turbo API. Since June, I have requested the result through function calling, so I get JSON that conforms to the schema instead of a paragraph with JSON embedded in it, which reduced my parsing to zero. The rule from 2021 has not moved by so much as a comma: generated output is input for review, not a fixture. The approved file goes into the repository, and that is what runs in CI. The model is not called during test runs, and I will return to this when discussing triggers.

I draft test cases in chat, just as I did in January, only more efficiently because I have established prompts and know what I cannot get out of it. One thing requires precision here, because in September 2023 it is easy to write something untrue. I have chat in the editor through the technical preview that I signed up for in March and received in spring. It is still a preview, not a feature that every Copilot subscriber can turn on in their settings. Organizations on the Business plan received a public beta of chat in July, but that is a separate path and a separate plan, not my individual account. I am writing this because I see texts online suggesting that chat “is already available to everyone,” and it simply is not, at least not on the day I am publishing this.

Now for the core of this section. All three things above produce input for the layer that was the cheapest in the entire suite: a unit test runs in milliseconds, needs no network, needs no browser, and does not flake. If a tool lowers the cost of producing tests in the layer that was already the cheapest to execute, the logical consequence is not fewer E2E tests. The consequence is a broader bottom, because suddenly it is worthwhile to write twenty boundary cases for an amount parser that were not worth an afternoon before. The pyramid becomes more of a pyramid, not less.

This does not mean I have less work. I moved it from writing to reading, and that is the entire transaction I have been making since January. Twenty generated boundary cases have to be read one by one because three of them will be duplicates, one will call Assert.NotNull on something that is never empty, and one will hit a business rule that does not exist in the demo.

There is no shortcut through the middle of the pyramid

For me, the middle consists of three things: contracts, integration on an in-memory host, and integration with a real dependency in a container. This is the layer where the model helps the least, and it is worth explaining why, because the reason is structural rather than temporary.

A contract test, which I described in the introduction to contract testing, is a record of an agreement between two teams. The model can generate the interaction skeleton and even guess field names correctly because it has seen an API like this a thousand times. It cannot do the one thing that gives this test meaning: it cannot agree with the other party that this is exactly what the response should look like, and it does not bear the consequences when the provider changes it. A pact is a document with two signatures. A generated pact without a conversation is a guess written in a JSON file, which is exactly the kind of artifact that looks like a process but is not one.

It is similar with in-memory host tests, which I described when writing about integration tests in .NET. Copilot completes the factory configuration and replacement of registrations in the dependency injection container quite efficiently because that is ceremonial code. The value of this test, however, lies in choosing what I replace and what I do not, and that choice comes from knowing what is risky in this application. The model does not have access to that knowledge because it is not in the code.

Containers remain exactly where they were, and for the same reasons as last year. A cold database start still takes several dozen seconds, and no model shortens that because it is the startup time of a process in an image, not my writing time. I documented the mechanics when writing about Testcontainers and Playwright in a pipeline, and after a year of AI, I have no corrections to make to that post.

There is one area where the model has genuinely helped me in the middle of the pyramid, and it is not generating tests. When I get a red run with a long stack trace from the integration layer, I paste the log into chat and ask for hypotheses about where to start. This means working with an error dump, not production code, and I have to observe the same rule I have followed since January: I do not send someone else’s data or logs from a company pipeline. The accuracy of such hypotheses is roughly the same as that of a colleague who knows the framework but does not know my application, which means useful in the first round and useless in the third.

UI: component, E2E, and screenshot are three different costs

Above the middle, I have three things that are easy to merge into one sentence about “UI tests,” and I want to avoid that because they have completely different prices.

I tried component testing in Playwright in August, which is where the seventeen tests in the table came from. I mount a single React component in a real browser without starting the entire application and without logging in. There is one thing to remember in every sentence on this topic: the package is called @playwright/experimental-ct-react, and the word experimental in the name is not decorative. It is not GA, the API may break with a minor version change, so I keep an exact pin and upgrade it on a separate branch. This is not a layer I recommend today as the foundation of someone’s strategy. It is a layer I am testing on my own demo.

Browser E2E stays where it was and still costs the same. Twenty-four scenarios, a full matrix of three engines, and wall-clock time measured in tens of minutes. Neither chat nor mounting a component removes the one thing for which this suite exists: checking that login, session, redirect, and header work together, in a real browser, in the assembled application. Component tests do not see routing or authorization because, by definition, they mount one piece.

Screenshot comparisons remain at nine and are deliberately not growing. I laid out the tool matrix when writing about visual regression tools, and I have not invalidated any of those conclusions. A toHaveScreenshot snapshot answers a question that neither CT nor E2E asks: does this still look the same? It is also the most sensitive to fonts, animations, and browser versions, so every added screenshot is a future maintenance cost, not a free assertion.

I find it convenient to write this down as three questions because it makes clear that none of them replaces the others. CT asks whether a component behaves correctly in isolation. E2E asks whether the path works when assembled. A screenshot asks whether the appearance has shifted. The model speeds up writing each of these three and does not change a single one of these three answers.

The triggers stay unchanged

Last year, I divided the suite into three baskets by trigger and described this in my CI/CD test strategy: a pull request takes what is fast and deterministic, main takes integration with real dependencies, and the nightly run takes full E2E and the matrix. After nine months with AI, I am not changing a single line in this division, and that is probably the strongest conclusion of this post.

The new unit tests go into the first basket because they are fast and deterministic, and that is the only reason. The fact that they were created with the model’s help gives them neither a speedup nor relief from requirements. As a result, my pull request gate grew from seven minutes to nine, and I consider those two minutes well spent because in return I got 117 assertions that simply did not exist before.

I put the component tests into the second basket, meaning on main after the merge, not in the gate. Not because they are slow, since they are not. Because the package is experimental, and I do not want a minor version change in someone else’s package to block someone’s merge. When it stops being experimental, I will move them, and that will be one sentence in the YAML file.

The model does not get its own job or its own step in the pipeline. I will repeat the argument from 2021 because it remains the only one I need: two identical calls do not have to produce the same result, so a step that depends on the model’s response is flaky by definition, and a red run starts to mean “someone else’s API had a bad day” instead of “the application is broken.” In April, I recorded the same decision for a generated pull request description, and I see no reason to reverse it. Generated text does not execute code, so it cannot be a gate.

There is one more temptation worth naming because I see it in myself. When a tool is impressive, I want to give it a place in the pipeline as proof that I appreciate it. That is the order backwards. A place in the pipeline belongs to something that answers a question that cannot be asked otherwise and answers it the same way every time. The model does not meet the second condition, and it does not meet the first because its product, the test code, is already in the repository and runs with the regular runner.

Summary

After nine months, the conclusion fits into one sentence: AI sped up the production of tests and did not change their distribution.

I am breaking this down into three sentences that I want to be able to read a year from now. The bottom of the pyramid grew by 117 unit tests, and the entire suite by 160, because writing them became cheaper, so the E2E share fell from 8 percent to 5 percent without removing a single E2E scenario because of the model. The middle, meaning contracts and integration, barely sped up at all because its cost is not writing but agreeing with the other party and waiting for real dependencies. The top costs the same as it did a year ago and runs on the same trigger because a nightly browser matrix did not become faster just because my editor suggested an assertion in it.

What does not follow from this? It does not mean I have a new layer called “LLM tests” because I do not have one and do not see a place for one. It does not mean everyone has chat in the editor today because mine is still a preview from the March waitlist, not a feature enabled through an individual account’s settings. It also does not mean that component testing is ready for the entire organization because the package has experimental in its name, and that says it all.

The shortest version of what I have understood this year goes like this: AI is a production line, not a new tier. It changes the pace at which a test is created and does not change the cost of executing it because that cost lies in the browser, the container, and the network, not in my writing. The pyramid has always been a diagram about execution cost and feedback speed. That is why it stays.

So I return to the question I have asked about every new tool since 2019: where does it fit in the entire testing process? A hundred and seventeen new unit tests do not fix a process in which no one reads a red result on a pull request. They only give me more things to read when that result finally detects something. In three months, I will write the year-end summary and then check whether the proportions in the table moved at all or whether all the rows merely grew together.