Testcontainers on Azure Pipelines

April was local - November is CI

In April, I described how I start a database from the test code. That entire post took place on my machine, where Docker is simply available and docker version responds immediately. The question I have received most often since then is different: fine, but will it work on CI?

This post answers that question and only that question. I do not repeat the fixture code here, show how to build the database container a second time, or explain why I should start a dependency from the test in the first place. All of that is under the link above. Here, the main characters are the azure-pipelines.yml file and the Docker daemon on the agent.

One thing has changed on the library side since April and is worth noting: DotNet.Testcontainers has a stable 1.5.0 release (July 2021). The package is still called DotNet.Testcontainers, the repository is HofmeisterAn/dotnet-testcontainers, and this is the version I use for this run. I pin it directly in the project file:

<PackageReference Include="DotNet.Testcontainers" Version="1.5.0" />

No range, no wildcard. If an integration test fails on CI, I want to know whether my code changed or the library underneath it changed. With a wildcard version, I can never be certain.

One more note at the start, so there is no ambiguity. .NET 6 was released a week ago. This is not a post about migrating the SDK, and I deliberately do not switch the project to version six just to make this run. I stay with what I already have configured.

Microsoft-hosted ubuntu-20.04

The cheapest version of this whole setup looks like this: Docker is installed and running on a Microsoft-hosted Linux agent, so DotNet.Testcontainers has something to talk to. I do not have to install anything extra, start a service, or work around user permissions.

I explicitly select ubuntu-20.04, not ubuntu-latest. Today they are the same image because ubuntu-latest already points to 20.04 after the migration around the turn of 2020 and 2021. But latest is a moving pointer in exactly the same sense that latest is a moving pointer for an image tag. Over time it will move to subsequent LTS releases, and it will do so without my involvement. I would rather change one line deliberately than explain to the team why the pipeline started failing on Monday morning.

Before I build anything, I make the agent tell me what I am working with:

- script: |
    docker version
    docker info --format '{{.OperatingSystem}}'
  displayName: What the agent has

This step costs one second and saves an hour of guessing. In the run log, I see the Docker client and server versions and the daemon’s operating system. I do not write down a specific Engine number here as a fact because the agent image is refreshed regularly, and the version I have today may not be the one you see in two weeks. What matters is that this line appears at all, because that means the daemon is responding. If the step ends with a socket connection error, there is no point in reading further into the test stack trace because the entire problem is here.

docker info filtered by operating system gives me a second important piece of information: this is a Linux daemon, so the images started by the test must also be Linux images.

Minimal azure-pipelines.yml without a container job

This is the whole file, without shortening it. One job, on the agent host, with no container: section:

trigger:
  - main

pr:
  - main

pool:
  vmImage: ubuntu-20.04

variables:
  buildConfiguration: Release

steps:
  - task: UseDotNet@2
    displayName: SDK
    inputs:
      packageType: sdk
      version: 5.0.x

  - script: |
      docker version
      docker info --format '{{.OperatingSystem}}'
    displayName: What the agent has

  - task: DotNetCoreCLI@2
    displayName: dotnet restore
    inputs:
      command: restore
      projects: "**/*.csproj"

  - task: DotNetCoreCLI@2
    displayName: dotnet test
    inputs:
      command: test
      projects: "**/*Tests.csproj"
      arguments: "--configuration $(buildConfiguration) --logger trx --results-directory $(Agent.TempDirectory)"

That is the entire mechanism. There is no step that starts the database, no docker run before the tests, and no docker-compose up. The test starts the container, just as it does locally, and the pipeline’s only role is to provide it with a working daemon and SDK.

It is worth noting what is not here. There is no Docker@2 or DockerInstaller@0 task. The Docker@2 task is used to build and push images in a pipeline, and I do not build any image. I only need the socket that is already there.

I run the dotnet test step through DotNetCoreCLI@2, not a plain script, mainly because I get result publishing for free. I will return to that at the end.

Why windows-2019 will not run a Linux image

This is the first thing I stumbled over when I tried to fit these tests into an existing pipeline that ran on Windows.

It is tempting to simply change vmImage to windows-2019 because Docker is also available there. The problem is that the Docker daemon works in one mode at a time. On a hosted Windows agent, Windows containers are the default mode, while the database image started by my test is a Linux image. As a result, the pull ends with a message about the image platform not matching the host platform. This is not a matter of Testcontainers configuration or package version, but of what is on the other side of the socket.

You do not see this problem on your own machine with Docker Desktop because switching between Windows and Linux containers is one click in a menu, and most of us have been using Linux mode since installation. I do not have that click on the agent, and I do not want to build a pipeline around switching the daemon mode.

The conclusion is short: a job that starts Linux containers from a test must run on a Linux agent. If my product builds on Windows, I split it into two jobs in the same pipeline - the build where it has to run and the integration tests on ubuntu-20.04. This is much simpler than fighting the image platform.

Container job vs Testcontainers: socket, mapDockerSocket, localhost

The second pitfall is more interesting and cost me more time.

Azure Pipelines lets me run an entire job inside a container. A container: section is enough to give me a repeatable environment with a specific SDK version, independent of what happens to be installed in the agent image:

pool:
  vmImage: ubuntu-20.04

container:
  image: mcr.microsoft.com/dotnet/sdk:5.0

steps:
  - script: dotnet test --logger trx
    displayName: dotnet test in a container job

It looks harmless, and in many projects this is a sensible pattern. But you have to understand what happens then because this is not nested Docker. A second daemon does not start inside my job container. The agent mounts the host socket, /var/run/docker.sock, into it, which lets the Docker client inside control the daemon running next to it on the host. The containers started by my test are siblings of the job container, not its children.

This has two consequences that need to be kept in mind.

First, it works as long as the socket is mounted. The YAML schema has a mapDockerSocket property, and setting it to false disables this mount. If someone on the team adds this line for security reasons, the tests will stop running with a daemon connection error, while the cause will be several levels removed from where the failure occurs.

container:
  image: mcr.microsoft.com/dotnet/sdk:5.0
  mapDockerSocket: false # this breaks Testcontainers

The second consequence is more subtle because nothing fails at startup. By default, Testcontainers constructs the container address from localhost and the port published on the host. This assumption is true when the test runs on the host. In a container job, the localhost of my test process is the network namespace of the job container, not the host, so the port published by its sibling is somewhere else. The test gets through container startup, then hangs while connecting and ends in a timeout that looks like a flaky test but is not one.

There are ways around this: put both containers on a shared user-defined network and connect by container name instead of localhost (1.5.0 added network support to the builder), or target the host gateway address instead of localhost. Both approaches work, and both require someone on the team to remember why the connection string looks different on CI than it does locally.

That is why my recommendation is unambiguous: for tests with Testcontainers, I stay with the path from the previous section, where the job runs on the agent host. A container job is a good tool, but in this particular combination it adds a network layer without giving me anything in return. I can pin the SDK version with the UseDotNet@2 task, and that was the only reason I reached for container: in the first place.

As an aside, hosted runners from another CI provider use exactly the same model with Docker on the host, so the reasoning carries over. This post is about Azure Pipelines, however, and that is where I will stay.

Pull time, pinning tags, cleanup

Locally, I pull the database image once and then only use it. A hosted agent works differently: I get a fresh machine, so every run is a cold start and pulls the image again. This is not a failure. It is the cost of entry, but it has to be accounted for instead of coming as a surprise.

I measure it from the log rather than guessing. A separate step before the tests shows me how much of the total job time is spent downloading the image itself:

- script: |
    date -u +"pull start: %H:%M:%S"
    docker pull $(dbImage)
    date -u +"pull end: %H:%M:%S"
  displayName: Pull the database image

This step is not required for the tests to work because the library will pull the image itself when starting the container. It is required for a conversation with the team when someone asks why the pipeline takes longer than before. With this number separated out, I know whether to look for savings in the tests or in the image.

I pin the tag in the same way I pinned it in January with a plain docker run. The database image version lives in one place in my test configuration, and it goes into a pipeline variable so that the pull step and the test refer to the same thing. latest on CI is an invitation to a situation where a green run on Tuesday and a red one on Wednesday differ by something that is not in the repository history.

Cleanup on a hosted agent is effectively free because the whole machine disappears after the job finishes. This does not mean, however, that I can leave containers open in the test code. First, I run the same suite locally, where nobody deletes my machine. Second, sooner or later some runs will end up on a self-hosted agent, where abandoned containers remain on disk along with their occupied ports. A container must be closed by the same code that opened it, not by the cleanup crew at the end of the pipeline.

On a self-hosted agent, I also add a control step that always runs, including after failed tests:

- script: |
    docker ps -a
    docker container prune -f
  condition: always()
  displayName: What is left after the run

The docker ps -a entry in the log is more important here than prune because it shows me whether I have a leak at all instead of silently sweeping it away.

And one thing this post does not fix. Integration tests with a real database can be unstable because of a race between writes and reads, and that problem is independent of where the container runs. I described it separately when discussing a retry policy. If you see random red runs after moving to CI, check first whether this is the pattern before you start blaming the agent.

I leave Publish Test Results as a familiar ADO mechanism

I deliberately do not make result publishing the second topic of this post.

The DotNetCoreCLI@2 task in test mode publishes results to the Tests tab on its own, which is why my YAML only contains --logger trx and the directory for the files. If I run the tests with a plain script, I add PublishTestResults@2 with the VSTest format and point it to the same .trx files, always with condition: always(), because I need the results from a red run more than from a green one.

This is an Azure DevOps mechanism that has worked the same way for a long time and does not change because the test starts the container instead of the pipeline. The Tests tab does not know and does not need to know where the database came from. To me, this proves that I moved Testcontainers to CI correctly: the report looks exactly as it did before, while the entire part of the configuration where the pipeline had to start dependencies manually has disappeared.

If I were composing the same set of dependencies declaratively alongside the tests, I would return to a Compose file. The difference is that the pipeline is then responsible for the environment lifecycle, while with Testcontainers the test is responsible for it. Both approaches are valid, but you have to choose one instead of using both at once.

One final reminder that I repeat to myself every time I move something to CI: simply running tests on an agent does not improve quality by even one step. Quality improves when the result is fast, repeatable, and read by the team, which means the process around the tests. Docker on the agent is only there so that this result appears without asking anyone for access to a shared database.