Forty minutes on one agent
My regression suite has around two hundred tests today. Some hit the API, some run through the browser, and everything is on .NET 5 and NUnit 3.13.2. Locally, I run a selected class and get a result in several seconds. On CI, the entire run takes forty minutes because the agent takes the tests one by one and nobody is in a hurry.
Forty minutes is the threshold beyond which people stop waiting for green. They start merging “because it will probably pass” and only look at the result when someone reports that something does not work in the environment. The feedback loop stops existing not because the tests are bad, but because they arrive too late.
I want to shorten this time, and I immediately write down one boundary condition: I am reducing wall-clock time, not the number of red results. Parallelism immediately rewards a mess in isolation with flaky tests. If the suite takes ten minutes after I enable four jobs, but one test fails for no reason in one out of three runs, I have replaced a slow signal with fast noise.
As an aside: Selenium 4.0.0 was released two days ago, on October 13, I am still on Selenium.WebDriver 3.141.0, and this post is not about migrating the driver.
Three levels of parallelism
The first thing I had to organize in my head was this: “parallel tests” means three different things, and mixing them in one conversation ends with someone enabling a flag that does not do what they thought it did.
Level one is threads in one process. The test framework decides this: NUnit or xUnit. Tests run concurrently in the same application domain, so they share memory, open connections, and anything someone thought to declare as static.
Level two is multiple processes on one machine. This is managed by the test platform, meaning VSTest: it starts a separate test host per core, and the unit of distribution is an assembly. Processes do not share memory, but they share the machine: ports, the file system, environment variables, and local Docker.
Level three is multiple machines, meaning pipeline jobs. Here, almost nothing is shared anymore except external resources, such as a common database in the test environment. However, every job has a cost: agent time, a repeated restore, build, and docker pull.
Microsoft describes levels two and three in the Azure Pipelines documentation under “parallel testing” for the VSTest task and “Run any tests in parallel”. Level one lives in the framework documentation, and it is the one most often forgotten at the start.
These levels multiply rather than add up: four jobs with four framework threads each mean sixteen tests at once. Each level breaks in a different way, so I enable them one at a time, in the order above, and check the flake rate after each step.
NUnit vs xUnit: which one is parallel by default
The default behavior of these two frameworks is exactly opposite, and this is probably the most common trap when changing the runner.
NUnit does nothing in parallel until I ask it to: I can have eight cores and an idle pipeline, and the tests will still run one by one. I enable it with the [Parallelizable] attribute at the assembly, class, or method level:
// any file in the test project, for example AssemblyInfo.cs
using NUnit.Framework;
[assembly: Parallelizable(ParallelScope.Fixtures)]
[assembly: LevelOfParallelism(4)]ParallelScope.Fixtures means that test classes can run in parallel with one another, but tests within a single class still run sequentially. This is my first step in every project because it provides a reasonable gain and does not require every test in a class to be independent of its neighbor. ParallelScope.All also runs methods in parallel, and I only reach for it when I am sure about isolation. LevelOfParallelism sets the thread pool size; without it, NUnit uses the processor count, and the CI agent and my laptop do not have the same number of cores.
xUnit 2.4.1 works the other way around: test collections run in parallel from the start, without any attribute. By default, one class is one collection, so out of the box I have parallel classes and sequential tests within a class, which is what I get in NUnit after adding ParallelScope.Fixtures. The thread count is set in xunit.runner.json:
{
"$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
"parallelizeTestCollections": true,
"maxParallelThreads": 4
}And here is the migration trap that I once fell into myself. A suite written for NUnit without attributes is sequential by definition, so it is allowed to have a static WebDriver and shared state. Moving the same code to xUnit enables parallelism simply by changing the library: tests fail in random places, while the commit contains not a single line about threads. Going the other way is less painful, but just as confusing: someone moves to NUnit, adds parallel: 4 in the pipeline, and wonders why nothing got faster, because the framework is still waiting for [Parallelizable].
There is one rule to take from this: before I touch the pipeline, I need to know how many tests actually run at once in one process. I set that number deliberately instead of inheriting it from the library defaults.
One machine: VSTest and /parallel
The second level is VSTest starting multiple test hosts on one agent. It is controlled by MaxCpuCount in .runsettings or the /Parallel switch in vstest.console:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<RunConfiguration>
<MaxCpuCount>0</MaxCpuCount>
<TargetFrameworkVersion>net5.0</TargetFrameworkVersion>
<ResultsDirectory>./TestResults</ResultsDirectory>
</RunConfiguration>
</RunSettings>MaxCpuCount set to 0 means “as many hosts as logical cores”, while 1 disables parallelism. I pass the file to dotnet test --settings tests/ci.runsettings, and the same thing in an Azure Pipelines task looks like this:
- task: VSTest@2
inputs:
testAssemblyVer2: |
**/*.Tests.dll
!**/obj/**
runSettingsFile: 'tests/ci.runsettings'
runInParallel: true
platform: 'x64'
configuration: 'Release'One thing to avoid a surprise: with runInParallel: true, the task controls the number of hosts itself, and MaxCpuCount from .runsettings stops mattering. It is not worth tweaking both places at once.
Now the most important caveat in this section, because this is where a couple of hours are most often lost. The unit of distribution in VSTest is an assembly. If all tests live in one Shop.UiTests.dll, then runInParallel has nothing to distribute: I get one test host and exactly the same time as before. The gain appears with several test projects, one for the API, one for the UI, and one for the data layer. Within a single assembly, only level one, meaning [Parallelizable], makes things faster.
The second caveat: all this happens on one machine because VSTest does not distribute tests across agents. To get from forty minutes down to ten, I need four machines, not four processes on one.
Multiple agents in Azure Pipelines: slices
In Azure Pipelines, one line can multiply a job:
jobs:
- job: functional
displayName: 'Functional tests'
strategy:
parallel: 4
pool:
vmImage: 'ubuntu-20.04'
steps:
- task: UseDotNet@2
inputs:
packageType: sdk
version: '5.0.x'
- script: dotnet build --configuration Release
displayName: 'Build'
- script: bash tests/slice.sh
displayName: 'Slice $(System.JobPositionInPhase) of $(System.TotalJobsInPhase)'
- task: PublishTestResults@2
condition: always()
inputs:
testResultsFormat: 'VSTest'
testResultsFiles: '**/*.trx'
mergeTestResults: truestrategy: parallel: 4 gives me four identical jobs, which means each one wants to run the entire test suite. That is the trap: multiplying the job does not make anything shorter, it only does the same thing four times. I have to perform the distribution myself, and Azure gives me two variables for this: System.JobPositionInPhase, with the job number from 1 to N, and System.TotalJobsInPhase, with the number of jobs.
My tests/slice.sh looks like this:
#!/usr/bin/env bash
set -euo pipefail
INDEX="${SYSTEM_JOBPOSITIONINPHASE:-1}"
TOTAL="${SYSTEM_TOTALJOBSINPHASE:-1}"
PROJECT="tests/Shop.UiTests/Shop.UiTests.csproj"
# list of test classes: fully qualified names, no arguments, no method name
dotnet test "$PROJECT" -c Release --no-build --list-tests \
| sed -n 's/^ \(.*\)$/\1/p' \
| sed 's/(.*$//' \
| sed 's/\.[^.]*$//' \
| sort -u > all-classes.txt
awk -v i="$INDEX" -v n="$TOTAL" '(NR - 1) % n == (i - 1)' all-classes.txt > slice.txt
FILTER=$(awk '{printf "%sFullyQualifiedName~%s", (NR > 1 ? "|" : ""), $0}' slice.txt)
if [ -z "$FILTER" ]; then
echo "Slice $INDEX of $TOTAL is empty - nothing to run."
exit 0
fi
echo "Slice $INDEX of $TOTAL, classes: $(wc -l < slice.txt)"
dotnet test "$PROJECT" -c Release --no-build \
--filter "$FILTER" \
--logger "trx;LogFileName=slice-$INDEX.trx"Four decisions here are deliberate, and each one follows from something that broke for me.
I split by class, not by individual test. dotnet test --list-tests prints full names together with test case arguments, meaning Shop.UiTests.CartTests.AddsProduct(1,"PLN"), and putting that name into --filter FullyQualifiedName=... ends in an argument about parentheses and commas. I therefore remove the arguments and the method name, and filter with the ~ operator, meaning “contains”. The side effect is useful: the class stays in one slice, so [OneTimeSetUp] runs once rather than in every job.
The filter is short as a result. Two hundred full names in one command line amount to more than a dozen kilobytes of arguments and create a real risk of truncation; a dozen or so class names fit without a problem.
An empty slice is not an error. With three classes and four jobs, one job has nothing to do and must finish green instead of failing because there are no tests.
Round-robin distribution by class is dumb but predictable. It distributes the number of classes evenly and knows nothing about execution time, so if one class takes eight minutes and all the others take half a minute each, that one class determines the wall-clock time. This is why I inspect timings from the .trx files and move heavy classes manually instead of adding another job.
There is an alternative: the VSTest@2 task can slice tests itself. It has distributionBatchType for this, with three modes: by the number of test cases, by execution time from previous runs, and by assembly. The time-based mode solves exactly this problem of uneven classes and is better than my awk. I leave it to the task when the tests are regular assemblies that it detects itself and I want automatic distribution. I filter manually when I want the split to be explicit in the repository, when I split by category instead of count, or when I run something that VSTest does not detect as a test.
GitHub Actions: strategy.matrix and max-parallel
In GitHub Actions, the same idea is expressed as a matrix:
name: functional-tests
on:
pull_request:
push:
branches: [main]
jobs:
tests:
runs-on: ubuntu-latest
strategy:
fail-fast: false
max-parallel: 4
matrix:
shard: [1, 2, 3, 4]
steps:
- uses: actions/checkout@v2
- uses: actions/setup-dotnet@v1
with:
dotnet-version: '5.0.x'
- run: dotnet build --configuration Release
- name: Tests - slice ${{ matrix.shard }}
env:
SYSTEM_JOBPOSITIONINPHASE: ${{ matrix.shard }}
SYSTEM_TOTALJOBSINPHASE: 4
run: bash tests/slice.sh
- uses: actions/upload-artifact@v2
if: always()
with:
name: trx-shard-${{ matrix.shard }}
path: '**/TestResults/*.trx'The same slice.sh handles both pipelines because it receives the slice number and count through environment variables with Azure names. Ugly, but I have one script to maintain, and I can run it locally by setting the same two variables.
fail-fast: false is mandatory here. By default, the matrix cancels the remaining jobs after the first failure, which makes sense when building on five platforms but is harmful for tests: I want to know whether one test failed in one slice or twenty failed across four, because these are completely different diagnoses.
max-parallel limits how many matrix jobs run at the same time. It is useful when there are more slices than resources, or when the tests hit a shared environment that cannot handle twenty clients. The upper limit for a matrix is 256 jobs per workflow run, and I have never come close to it.
If the tests are already tagged, instead of numbered slices I can distribute the matrix by category, meaning suite: [smoke, cart, checkout, admin] plus --filter "TestCategory=${{ matrix.suite }}", which for NUnit corresponds to the [Category("cart")] attribute. I like this variant for its readability because I see names rather than numbers in the results. The drawback is the uneven size of categories, which nobody balances automatically.
I also want to clear up one misunderstanding separately because I have already heard it twice. The concurrency key, which GitHub added in April this year, has nothing to do with distributing tests. It limits overlapping workflow runs or jobs in one group and can cancel the one in progress. I use it so that two pushes in a row do not start two deployments to the same environment:
concurrency:
group: deploy-${{ github.ref }}
cancel-in-progress: trueThis is a safeguard against too many runs, not a slicing mechanism. Only matrix is responsible for the slices.
One more small note about the runner image: ubuntu-latest is an alias and points to 20.04 today, but one day it will move without my involvement. Where reproducibility matters to me, I specify the version directly, just as I have pinned image tags since January in Docker for QA.
Isolation or flakes
This is the real substance of this post. The YAML above is twenty lines, while what follows took me a week of work.
A static WebDriver is killer number one. private static IWebDriver _driver passes all sequential tests and breaks immediately after parallelism is enabled because two classes control one browser, and whichever clicks first wins. The remedy is an instance field plus a fresh class instance for every test, which NUnit has supported since version 3.13 in January:
using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
[TestFixture]
[Parallelizable(ParallelScope.Self)]
[FixtureLifeCycle(LifeCycle.InstancePerTestCase)]
public class CartTests
{
private IWebDriver _driver;
[SetUp]
public void SetUp()
{
var options = new ChromeOptions();
options.AddArgument("--headless");
options.AddArgument("--window-size=1280,900");
_driver = new ChromeDriver(options);
}
[TearDown]
public void TearDown()
{
_driver?.Quit();
_driver?.Dispose();
}
}LifeCycle.InstancePerTestCase means NUnit creates a new test class object for every test, so instance fields do not leak between tests even by accident; the default is the opposite, meaning one instance for the entire class. One limitation is worth remembering: [OneTimeSetUp] and [OneTimeTearDown] must then be static, which is actually healthy because it immediately shows what is shared by the whole class.
This is the same idea I kept returning to when refactoring tests in Cypress: the test sets up its own state and does not assume that someone did it beforehand. Back then it was about readability; with parallelism, the same assumption determines whether the pipeline is green.
A shared port is killer number two. In May, I ran WireMock as a mock for a partner API on a fixed port 58090, and in the Compose environment I pinned the database and cache in the same way. As long as one environment is running on the agent, everything is fine; with two jobs on the same machine, the second gets “address already in use”. A container started for a job therefore needs its own ports and its own data, not the values from the README. I offset the ports by the slice number and give the Compose project a separate name:
export COMPOSE_PROJECT_NAME="shard-${INDEX}"
export MOCK_PORT=$((58090 + INDEX))
export DB_PORT=$((55432 + INDEX))
docker-compose up -dThe project name is more important here than it looks: without it, Compose decides this is the same project, and the second job takes over the first job’s containers. When I want complete certainty about ports, I reach for the approach from April, meaning Testcontainers with a port mapped by the library, and ask the container where it started.
Killer number three is a shared database. Two slices relying on the same test account or the same lookup table will delete data from under each other. There is no magic flag in YAML here, only a decision: either each job gets its own database or schema, or tests create their own data with a unique prefix and clean up only after themselves. Queries such as DELETE FROM Orders in [TearDown] stop being acceptable the moment I enable the second job. The same applies to directories: screenshots, .trx files, and the download directory get the slice number in their names.
If browsers rather than tests need to be distributed, there is a second axis for that, meaning Selenium Grid in Docker from March. However, when every job starts its own headless Chrome locally, Grid is no longer needed.
And one sentence I need to say directly, because I am tempted myself: retrying is not isolation. In July, I described retry policies with Polly, and I still use them, but they are for transient network and external service failures. Wrapping a retry around a test that lost a race with its neighbor over a database row fixes nothing. It turns a red result into green on the second attempt, which removes the only signal that two tests share state they should not share.
How much it costs
Parallelism is not free, and this is not only about the bill.
First, the number of jobs that actually run at the same time depends on the account’s plan. A matrix with twenty slices and a lower parallel job limit will queue up, and the wall-clock time will decrease only by the number of jobs that really start together. The specific numbers need to be checked in the documentation for the plan on the day they matter because this is something that changes.
Second, parallel jobs are a separate resource in Azure Pipelines. For new organizations, the free allocation stopped being granted automatically this year: the change took effect for public projects in February and for private projects in March, and an application has to be submitted. Organizations that already had the allocation were unchanged. The practical conclusion: parallel: 8 in a new organization can mean eight jobs queueing for one available slot, and I will not see any error in the log, only a slow run.
Third, every job has a fixed cost: checkout, SDK installation, restore, build, and docker pull. With four jobs, ten minutes of tests plus four minutes of warm-up is a great deal. With sixteen, I have two and a half minutes of tests and still four minutes of warm-up, meaning I pay for sixteen machines to get from fourteen minutes down to six. This is why, before adding another slice, I reduce the fixed cost: build once in a separate job, publish the binaries as an artifact, and have the test jobs only download them and call dotnet test --no-build. NuGet package caching is available in both systems.
Summary
The order matters more than the tools themselves here, so I am writing it down at the end.
Isolation first: no static state, no fixed port, no shared database between slices. Without this, everything else produces flakes, only faster. Then the framework level, because it is the cheapest: [Parallelizable(ParallelScope.Fixtures)] in NUnit with an explicitly set LevelOfParallelism, or consciously accepting what xUnit does by default. Then multiple processes on the agent, if I have several test projects, because only then does runInParallel have something to distribute. Finally, multiple jobs: strategy: parallel with slices based on System.JobPositionInPhase in Azure, or strategy.matrix with fail-fast: false in GitHub Actions.
And two numbers that, starting today, I record after every such exercise: the wall-clock time of the entire run, because it determines whether anyone still waits for the result, and the percentage of runs in which something failed without a code change, because it determines whether that result means anything. The number of jobs is not a metric, it is a cost.
For me, it went from forty minutes down to eleven, with four jobs and parallel classes within each one. Distributing the work across jobs provided the biggest time gain, but the boring work from the isolation section provided the biggest gain in confidence in the suite: removing the static driver and separating the data. After that, I stopped seeing red tests that were green after another run.
I have one thing left to organize, and it is already visible in this post: data. Prefixes, cleaning up after each test, and separate accounts per slice are workarounds for not having a written test data strategy, and with parallelism this stops being a matter of hygiene. I want to describe it separately. It all still comes back to the question I ask myself every time: where does this fit into the entire test process? A faster pipeline does not fix the process. It only makes what is not working in it visible sooner.