Docker for QA - on-demand test environments

Why a tester needs Docker

The most frustrating part of my work with integration tests is not writing assertions. It is waiting. Waiting for someone to free up the shared database, for a deployment to the shared environment to finish, or for a colleague to stop testing a migration on the same instance where I want to run my suite. A shared test environment has one advantage - there is only one, so it is easy to describe. And it has one disadvantage that cancels out that advantage: there is only one, so everyone fights over it.

When I wrote Environment preparation, I installed dependencies directly on my machine. That works exactly until I need two versions of the same database or want to return to a clean state without uninstalling half the system.

That is why I am increasingly taking the opposite approach: I spin up the dependency I need for a test locally for a while, then kill it after the run. That is the entire idea behind this post. No orchestration, no cluster, no managing a container from the test code. One command, one dependency, one cleanup.

What I have installed

Before I run anything, I check what I am working with:

$ docker version --format '{{.Server.Version}}'
20.10.2

The 20.10 line is new. Engine 20.10.0 was released on December 9, 2020, and 20.10.2 has release notes dated January 4, 2021. On Windows and macOS machines, I do not install Engine separately. I use Docker Desktop instead - version 3.1.0 was released just yesterday, January 14. It is worth remembering that the Desktop version and the Engine version inside its virtual machine are two different things. I do not guess. I read what docker version shows on my machine, and I give that version to the team when someone cannot reproduce my run.

If the command above returns nothing, the daemon is not running and there is no point in continuing with the rest of this post. On Linux, I check the service. With Desktop, I simply check whether the whale icon is green.

A minimal docker run

Most often, I need a database. I take the official image and pin a specific tag:

$ docker run -d \
    --name qa-postgres \
    -e POSTGRES_PASSWORD=test \
    -e POSTGRES_DB=shop \
    -p 55432:5432 \
    postgres:13.1

Several things in this command are deliberate.

postgres:13.1, not postgres:latest. latest is not a version. It is a moving pointer. If the test passes for me in January but fails for someone else in March, I want to be sure that we both had the same engine. Pinning the tag is the cheapest thing I can do for repeatability.

-p 55432:5432, not -p 5432:5432. The port on the left is on my machine, and the port on the right is inside the container. If I have PostgreSQL installed locally, the standard port is already in use and the container will not start. I deliberately choose a high, unusual port so I do not have to guess later what my test is connecting to. The connection string in the test configuration then points to localhost:55432.

--name qa-postgres. Without a name, Docker will invent a random one, and I will copy an identifier for every subsequent command. With a name, I have a stable handle for logs, exec, and removal.

-d detaches the container from the terminal. When I debug startup, I run it without -d and watch the output live.

If I need the database for only one run and do not care about the data, I add --rm so that the container disappears after it stops, and I keep the data directory in memory:

$ docker run -d --rm \
    --name qa-postgres \
    -e POSTGRES_PASSWORD=test \
    -p 55432:5432 \
    --tmpfs /var/lib/postgresql/data \
    postgres:13.1

This container writes faster and is clean by definition on every start. I do not use this variant when I want to inspect the data after a failed test.

Ports, logs, and readiness

The most common mistake I made at first was starting the test immediately after docker run. The container is running, but the process inside is still initializing, so the connection is refused. docker run finishes when the container has started, not when the application inside it is ready.

The first thing I check is whether the container is running at all and which ports it exposes:

$ docker ps --filter name=qa-postgres
CONTAINER ID   IMAGE           STATUS         PORTS                     NAMES
2f1c9a7d5e10   postgres:13.1   Up 8 seconds   0.0.0.0:55432->5432/tcp   qa-postgres

If the container is not here, I add -a and check the exit code, then read the logs:

$ docker logs qa-postgres

Logs solve most problems: a bad volume, a missing required environment variable, or a port conflict. It is worth getting used to them before blaming the test.

I check readiness with a tool included in the image:

$ docker exec qa-postgres pg_isready -U postgres
/var/run/postgresql:5432 - accepting connections

This is exactly the part worth saving in a script because I repeat it every day.

A short startup script

I do not build a framework around this. One file is enough. I put it in the test repository next to the configuration file:

#!/usr/bin/env bash
set -euo pipefail

NAME=qa-postgres
IMAGE=postgres:13.1
PORT=55432

# If the previous run left something behind, start from a clean state
docker rm -f "$NAME" > /dev/null 2>&1 || true

docker run -d \
  --name "$NAME" \
  -e POSTGRES_PASSWORD=test \
  -e POSTGRES_DB=shop \
  -p "${PORT}:5432" \
  "$IMAGE" > /dev/null

for _ in $(seq 1 30); do
  if docker exec "$NAME" pg_isready -U postgres -d shop > /dev/null 2>&1; then
    echo "Database ready on localhost:${PORT}"
    exit 0
  fi
  sleep 1
done

echo "Database did not start within 30 seconds. Logs:"
docker logs "$NAME"
exit 1

The script does three things: removes the previous container with this name, starts a new one, and waits for readiness instead of blindly using sleep 10. Its exit code is honest, so I can run it before the tests and stop the run if the dependency does not start. A tester who receives a repository like this does not need to know anything about Docker except that it must be installed.

Cleanup

This part is more important than it looks. Abandoned containers are the most common cause of the reverse version of “it works on my machine”: it does not work on mine because I still have yesterday’s container using the same port or containing data from a failed migration.

$ docker stop qa-postgres
$ docker rm qa-postgres

Or in one step when I no longer care about the container:

$ docker rm -f qa-postgres

From time to time, I review what remains:

$ docker ps -a
$ docker volume ls

A container removed with docker rm does not automatically delete the data volume created by the image. If I see a list of anonymous volumes after a week, it means that each of my runs left something behind. docker rm -v removes volumes associated with the container. I perform bulk cleanup deliberately and after checking the list because prune does not ask a second time.

My rule is simple: a test container should live as long as the test run. If I have to keep it longer, it is no longer an on-demand environment. It has become another small server that needs looking after. That is exactly the problem I was trying to escape.

What this post does not cover

I am deliberately leaving two things for later.

The first is running many services at once. When I need a database, a broker, and a mock at the same time, manual docker run commands stop being convenient, so I move them to a declarative file run by docker-compose (I have version 1.27.4 pinned locally). That is a separate topic with its own pitfalls around networking and startup order.

The second is managing the container lifecycle from the test code, so that the test itself, rather than a script, decides what to start and when. It is an interesting direction that I want to try, but I do not yet have results on which I could base a post.

Before I automate anything more deeply, however, it is worth taking a step back and looking at the whole picture: what the test process should look like. Docker does not improve quality by itself. It only shortens the feedback loop, and that is still a great deal.