A green PR nobody waits for
Almost a year ago, I described how I cut a forty-minute regression run down to eleven minutes by splitting the suite across four jobs. That number was real, and the mechanism still works. The problem is that over the next twelve months, I added more things to the same run: a database container instead of a shared instance, a partner mock instead of waiting for their environment, contract verification, and a second browser in the UI suite. Each of these added a few minutes on its own, and each was worth the cost.
I added it all up last week because I wanted to know exactly what I was waiting for after pushing a branch. My required pull request checks take thirty-four minutes. I am exactly where I was trying to escape from a year ago, only at a higher cost: I have more jobs, more containers, and the same feedback loop.
The symptoms are always the same, and they have nothing to do with the machines. Nobody sits and watches for thirty-four minutes. People switch to another task, return an hour later, see one red job, and click “re-run” without reading the log because “it is probably that test again.” If it is green the second time, nobody checks whether the first run was telling the truth. A test suite treated this way stops being a gate and becomes a formality.
Adding another four jobs is tempting because it takes one line in YAML. This time, I do not want to follow that path. Slicing the suite reduces wall-clock time in proportion to the number of machines, but only until the fixed cost of a job starts consuming the gain. I did the math a year ago, and nothing about that arithmetic has changed. That leaves the second question I did not ask then: does this test need to run before the merge at all?
This post is about that question. It is not about tools because I have already chosen them: .NET 6, xUnit 2.4.2 from August 1 or NUnit 3.13.3, containers, Compose, and browsers. They all stay. I am changing only which trigger runs each one.
Three buckets: PR, main, nightly
I listed everything CI runs today and divided it into three buckets using one criterion: what happens if this test fails ten minutes after the merge instead of ten minutes before it?
Bucket one, the pull request. This is where I put checks that can block a merge and do not lie. Build and analysis, unit tests, consumer-side contract tests, and one smoke test that checks whether the application starts at all and responds in one important scenario. Consumer contracts belong here because they are cheap: the test writes a pact file from a client-side mock and needs neither a network nor a live provider. The goal of this bucket is not “complete confidence” but minutes. If something does not fit into a few minutes and is not deterministic, it is not a candidate for a required check, even if it is the smartest test in the repository.
Bucket two, main after the merge. This is where I move everything that needs real dependencies: integration with a database in a container, a queue, provider-side contract verification, and a broader API suite. This run may take fifteen minutes, and it may sometimes be red. A red main is an event someone responds to immediately, but it does not make anyone wait with an open pull request. I consciously accept that a defective commit may sit on the main branch for a while. The alternative is making every author wait forty minutes after every typo in a README.
Bucket three, the nightly schedule. Full browser-based E2E, an engine matrix, slow scenarios with larger datasets, and long business paths. This suite provides the most knowledge about the product and is the worst fit for a gate: it is slow, runs through an interface that changes more often than contracts, and depends on environment state. The nightly run is not a required pull request check, and it is not supposed to be one.
A sketch of this division fits into three jobs. I deliberately omit checkout, SDK installation, and result publishing so that only the triggers are visible:
name: tests
on:
pull_request:
push:
branches: [main]
schedule:
- cron: '0 2 * * *'
jobs:
pr-gate:
if: github.event_name == 'pull_request'
runs-on: ubuntu-20.04
steps:
- run: dotnet test tests/Unit -c Release --no-build
- run: dotnet test tests/Contracts.Consumer -c Release --no-build
- run: dotnet test tests/Smoke -c Release --no-build --filter "Category=smoke"
integration:
if: github.event_name == 'push'
runs-on: ubuntu-20.04
steps:
# dependency containers start from the test
- run: dotnet test tests/Integration -c Release
- run: dotnet test tests/Contracts.Provider -c Release
nightly:
if: github.event_name == 'schedule'
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
browser: [chromium, firefox, webkit]
steps:
- run: npm run e2e -- --project=${{ matrix.browser }}Three things in this sketch do all the work, and none is a syntax trick. First, pr-gate is the only job I make required in the branch policy. The other two can be red without blocking anyone’s merge. Second, each bucket has a separate test project directory because a division that is not visible in the repository structure disappears after a month. Third, the nightly matrix has fail-fast: false because I want the complete picture from a nightly run, not just the first failure. I explained the matrix mechanics in parallel tests on CI and am only using them here.
The same division is just as concise in Azure Pipelines: pr and trigger in the YAML file, plus a separate pipeline with schedules. The names differ, but the decision is identical.
Speed versus confidence
One reframing helped me the most when I moved tests between buckets. Each layer has a different cost of a flake, and that is what determines where a test should live.
A flaky test on a pull request costs the most because it damages not just one run but a habit. Someone sees a red required check, knows from experience that half the time it is not telling the truth, and stops reading. From that point on, the gate no longer protects against anything because it has taught the team that its result is optional. That is why I let only deterministic checks into the first bucket and remove anything that passes “almost every time.”
The same flake in a nightly run costs one issue to investigate in the morning. I am happy to pay that price because, in return, I get a test that runs through the real interface and in the real environment. The nightly suite is allowed to be temperamental as long as someone looks at it in the morning.
This leads to a rule I pin up for myself because I am tempted to break it. A retry is not a pass from nightly to pull request. I described retry policies with Polly a year ago and still use them, but they are for temporary failures of the network and third-party services, not for masking a race for data. Wrapping a test in a second attempt does not change its cost or reliability. It only changes whether I see the problem. A test that needs three attempts to fit into the gate simply does not belong in the gate.
It is also worth stating plainly what a pull request does not prove. A green pr-gate does not mean “ready for release.” It means exactly this: the change does not break the unit tests, does not break the contract with consumers, and the application starts after it. The rest of the confidence comes later, and that is a conscious decision, not a gap I forgot to close.
What I am not repeating here
Three things in this post are already on the blog, and I do not intend to write them a second time.
Isolation and three levels of parallelism: framework threads, VSTest processes, and pipeline jobs. The order from that post still applies and is a prerequisite for everything discussed here. Splitting a poorly isolated suite by trigger produces exactly the same result as running a poorly isolated suite in parallel, which is faster flakes. I explained where test class and fixture state live in January when I wrote about organizing tests in xUnit and NUnit.
A container on the agent: Testcontainers in Azure Pipelines from November, including the socket and the choice between a container job and the host daemon. That post explains how. This one only says that heavy integration with a container belongs on main and nightly, not on every pull request.
A browser on CI: Cypress on Azure Pipelines. That also stays unchanged. One sentence changes: the full browser matrix is not a required pull request check.
Data remains a separate subject, and I still do not have it organized perfectly. I described test data management strategies in September last year, and that knowledge underpins the entire second bucket. Without separate data for each run, a main job with several parallel slices becomes a lottery.
Containers after a year
A year with Testcontainers and Compose changed one thing in the way I calculate costs. A container is not free just because it starts from code.
A cold start of a database in a container on a fresh agent still takes me a good few dozen seconds before the server begins responding, plus the time to pull the image if the agent does not have it. For one integration test, that is noise. On every pull request, it is a tax paid by everyone, including the author of a change to a configuration file. That is why integration with real dependencies sits in my second bucket: once per merge, not once per push.
The versions as of today, so the post does not drift away from reality: DotNet.Testcontainers 1.6.0 from June 7 on the main job, with containers started from the test just as in my first approach to MSSQL. Where I have several dependencies and want to describe them declaratively, Compose still does the job.
One note, without turning it into the subject of this post. Compose V2 has been generally available since April 26 this year, and I call it without a hyphen, as docker compose. My posts about Docker for QA and a test environment with Compose are from last year and teach the hyphenated variant. I am not rewriting them or forcing a migration. On a newly created job, I simply write version two and leave it at that. This is a command name change, not a strategy change.
UI in September
Browser-based tests are where dividing the suite into buckets produces the biggest gain because they are both the most valuable and the most expensive.
The rule is simple: the full matrix goes to nightly, while at most one scenario goes into the pull request. That one scenario has to meet a strict condition: its own state, its own data, and no dependency on what the previous run left behind. If a UI smoke test requires an account created in the environment last week, it is not a smoke test but a time bomb. Selenium Grid in Docker from last year remains a tool for the matrix, not for a single scenario on every push. Headless mode in the same job is enough for one smoke test.
In terms of tools, September finds me between options, and I do not pretend otherwise. The Cypress series on the blog describes a suite I actually run, and nothing is disappearing from it today. In parallel, I explained in the spring why I am starting to move away from Cypress, and I have an empty project set up in May. The package is at 1.25.0 from August 11 today. This does not matter much for this post, which is exactly the point: the division into buckets does not depend on which runner wins. Whatever I am using a year from now, the full browser matrix will still run at night, not at the gate.
Signal, not noise
Dividing by trigger only makes sense when every bucket has its audience. Otherwise, the nightly run becomes wallpaper after a month.
Publishing results is a requirement here, not decoration. Results from every run land as artifacts and a report, just as when publishing Cypress results, and the nightly run additionally sends a Slack message after the job finishes. The difference is in the content: from a pull request I care only about a red result, while from nightly I also care about a series of green runs because only against that series can I see which test is flaky.
The second rule concerns the list of required checks and is less obvious than it appears. A required check should be small and stable. When everything the pipeline can run is added to the branch policy, the list stops meaning anything: every red item is equally important, so none is important, and people choose which one to watch anyway. I therefore keep the list short and treat every addition to it as a process change, not a configuration change.
The third thing is the owner of the nightly run. Without a name on the calendar, this bucket dies in three weeks. There is no flag for this in YAML, only an agreement that someone looks in the morning and either opens an issue or removes the test.
Summary
The order is more important than the particular syntax, so I will record it at the end.
First comes isolation because, without it, everything else produces noise faster. Then comes the division by place in Git: the pull request takes unit tests, consumer contracts, and one smoke test; main takes integration with real dependencies and provider-side verification; nightly takes full E2E and the matrix. Only then comes parallelism, applied separately in each bucket because each one has a different budget.
I measure two numbers and deliberately do not combine them into one. The first is the wall-clock time of the required checks themselves because it determines whether anyone still waits for the result. The second is the percentage of nightly runs in which something failed without a code change because it determines whether looking at the report in the morning makes sense. Improving the first at the expense of the second is easy and always ends the same way.
After rearranging the buckets, my pull request gate dropped from thirty-four minutes to seven, while the total machine time did not decrease at all because the same tests still run, only elsewhere. That is the whole transaction: I did not make anything faster; I moved the waiting from a person to a schedule. This brings me back to the question I ask myself every time I work on the pipeline: where does this sit in the whole test process, and did I promise myself something different in last year’s summary? A faster gate does not improve quality. It only makes people read its result again.