Test data in .NET - factories, Respawn, containers

Three pains I am starting with

I promised myself in June that I would organize test data into a single strategy in the autumn instead of patching it wherever it happened to hurt. It is September, so I am sitting down to do it. I will start with three specific situations because they best explain the point of this post.

First. An integration test suite runs against a single database, and all tests share the same user. Test A assumes that the user has no orders yet, while test B has just added one. Run in sequence, they pass; run in a different order, they do not. When one stops halfway through, the next run starts with the database in a state nobody designed. The worst thing is that such a test is not permanently red. It is red every fourth time, so the team learns to click it again instead of reading it.

Second. Fixtures live in JSON files that somebody generated once, and they drift away from the model. A required field is added to the class, deserialization puts null into it, the test passes because its assertion does not touch that field, and only in the application does it become apparent that half the initial data is incomplete. A JSON file is not compiled with the project, so nobody gets a warning.

Third. An API mock returns a different shape than production. In May, I described how I run WireMock in a container, and that same post ended with an honest caveat: a stub is my idea of someone else’s response. When the stub lives in a directory on the CI agent’s disk rather than in the repository next to the test, that idea starts living a life of its own.

All three pains have a common denominator. It is not about the tool, but about nobody consciously deciding where the data comes from and who cleans it up. This post is an attempt to make that decision for .NET 5, with pinned versions, using variants that can be implemented today.

Strategy map: four layers of test data

I divide the problem into four layers. This is not a ranking. It is a list to choose from consciously, and most often several are used together in one repository.

The first layer is in-memory data built by a factory in test code. There is no database and no network; the object is created and disappears with the test. The second layer is a single shared database whose state is reset before every test. The database is running, the schema is ready, and I remove its data before the test. The third layer is a disposable database started in a container for a test or test class. The fourth layer has no database on the other side: an HTTP stub responds instead of the real service.

One column that is not and will not be in this table is “production copy.” I know why it is tempting. A dump has real distributions and real edge cases, and I do not have to invent it. It also has real personal data, real addresses, and real amounts that end up on the laptop of everyone who clones the repository. Anonymizing a dump is a separate project, not a step in a startup script, and until that project exists, all the examples in this post are synthetic.

Seeded factory: Bogus 33.1.1

I start with the cheapest layer because it solves the second pain on the list. A fixture in code is compiled together with the model. Adding a required field to the class breaks the factory build instead of remaining silent in a JSON file.

In .NET, I use Bogus for this. The latest version is 33.1.1 from August 29, so two weeks ago. Previously, I was on 33.0.2 from February.

$ dotnet add package Bogus --version 33.1.1
$ dotnet add package Respawn --version 4.0.0
$ dotnet add package WireMock.Net --version 1.4.20
$ dotnet add package DotNet.Testcontainers --version 1.5.0

I pin versions for the same reason I pin image tags: I want a run three months from now to get the same library, not whichever one happens to be released by then. The container package is still called DotNet.Testcontainers, and that is the name I search for on NuGet.

My factory looks like this:

public sealed class UserFactory
{
    private readonly Faker<User> _faker = new Faker<User>("pl")
        .UseSeed(42)
        .RuleFor(u => u.Id, f => f.Random.Guid())
        .RuleFor(u => u.FirstName, f => f.Name.FirstName())
        .RuleFor(u => u.LastName, f => f.Name.LastName())
        .RuleFor(u => u.Email, (f, u) => f.Internet.Email(u.FirstName, u.LastName, "example.com"))
        .RuleFor(u => u.CreditLimit, f => f.Finance.Amount(100m, 5000m))
        .RuleFor(u => u.CreatedAt, f => f.Date.Past(2, new DateTime(2021, 9, 1)));

    public User Build() => _faker.Generate();

    public List<User> Build(int count) => _faker.Generate(count);
}

Three decisions here are deliberate.

UseSeed(42) is the most important line in the entire class. Without it, Bogus generates a different set every time, so a test that passed yesterday can hit a name with an apostrophe or an amount with three decimal places today and blow up. With the seed set, the same code produces the same result on my laptop and on the agent. When I want more variants, I do not remove the seed. I provide a different one and record it explicitly in the test name.

The "pl" locale gives me first names, surnames, and addresses that look Polish. This is not cosmetic. A set generated with the default locale has no diacritics, and diacritics are exactly what breaks encoding in CSV exports and column widths in lists. If the application supports several markets, I keep one factory per locale.

Addresses end up in the example.com domain, which is a domain reserved for documentation. None of this will send anyone an email if the test environment configuration turns out to have a hole in it.

When a test needs one specific account that I assert by address, I do not leave it to chance:

public static User SeededUser() => new Faker<User>("pl")
    .UseSeed(42)
    .RuleFor(u => u.Id, f => f.Random.Guid())
    .RuleFor(u => u.Email, _ => "test.user+seed42@example.com")
    .RuleFor(u => u.FirstName, f => f.Name.FirstName())
    .RuleFor(u => u.LastName, f => f.Name.LastName())
    .Generate();

The address is ugly, and it is meant to be. It contains a plus sign so that I also test whether address validation rejects it, and it has the seed in the name so that if it appears in a log after a production mistake, it is immediately clear that this account came from tests.

I no longer use the names User1, User2, and User3 anywhere. Not because they are ugly, but because they say nothing. When a test fails on SeededUser or UserWithoutOrders, I know from the name alone what made that record different.

The alternative is AutoFixture, in my case version 4.17.0 from April. I use it when an object has no domain semantics and I genuinely do not care what is in the fields, only that they are not empty. A one-line fixture.Create<AddressDto>() saves me thirty lines of assignments in a DTO that the test does not inspect at all. Where data needs to look like data, I return to Bogus because AutoFixture will fill Email with a random GUID and will be right to do so, but that does not help me.

One sentence about a topic that has been hanging in the air since June: a Completions-based generator is a desk experiment for a dozen or so records reviewed by a person, not a strategy for an everyday test suite, and it does not occupy any of the four layers in this post.

Shared database: Respawn 4.0.0 and Checkpoint

A factory will not test mapping or migrations. That requires a real database, and with it the first pain returns: state remains after the test.

For years, I did this in the most brutal way, dropping and creating the entire schema before a test class. It works, and it is very slow. Migrations on a medium-sized database can take dozens of seconds, and I do all of that just to remove a dozen or so rows.

Respawn reverses this idea. Instead of recreating the schema, it leaves it alone and removes the data, first calculating the table order from foreign keys. In September 2021, the current line is 4.0.0, and the API revolves around the Checkpoint class.

public sealed class DatabaseFixture
{
    private static readonly Checkpoint Checkpoint = new Checkpoint
    {
        SchemasToInclude = new[] { "dbo", "sales" },
        TablesToIgnore = new[] { "Lookups", "Countries", "__EFMigrationsHistory" }
    };

    public string ConnectionString { get; }

    public Task ResetAsync() => Checkpoint.Reset(ConnectionString);
}

And its use in an xUnit test, still version 2.4.1 in my case:

public class OrderRepositoryTests : IClassFixture<DatabaseFixture>, IAsyncLifetime
{
    private readonly DatabaseFixture _database;
    private readonly UserFactory _users = new UserFactory();

    public OrderRepositoryTests(DatabaseFixture database) => _database = database;

    public Task InitializeAsync() => _database.ResetAsync();

    public Task DisposeAsync() => Task.CompletedTask;

    [Fact]
    public async Task New_user_has_no_orders()
    {
        var user = _users.Build();
        await _database.InsertAsync(user);

        var orders = await _database.GetOrdersAsync(user.Id);

        Assert.Empty(orders);
    }
}

I reset before the test, not after it. The difference is practical: when a test fails, I want to be able to inspect the database and see what remains there. Cleaning up after myself removes evidence from the scene.

TablesToIgnore is a list that needs to be thought through once and then maintained. Lookup tables, meaning countries, currencies, statuses, and VAT dictionaries, are not test data. They arrive through a migration or startup script, and deleting them means that they must be filled again after every reset, while foreign keys from the tables that use them break along the way. The Entity Framework migration history is here for the same reason: clearing it makes the application consider the database untouched and start migrating from scratch.

There is one limitation to know before implementing this. Respawn calculates the deletion order from foreign key relationships in the database. If those relationships are not declared because someone once “temporarily” removed a constraint during deployment, the library does not know about them, and the first reset ends with an integrity violation or, worse, a silent orphan in a child table. Cycles in the key graph can also hurt. This is not a flaw in the tool. It is a free schema audit, only performed at an inconvenient time.

Disposable database: seed after StartAsync

Respawn assumes that a database is running somewhere. When I do not want to make that assumption, I start it from the test. I described this in April when running MSSQL with Testcontainers, and I will not repeat the basics of the library itself here. I am adding one thing that post did not cover: what happens to the data after StartAsync.

The container starts with an empty instance. This means that after starting it, I need to do two things in a fixed order: create the schema and insert the initial data. The schema comes from application migrations, not SQL written by hand in the test, because otherwise the test checks a schema that does not exist anywhere else. The initial data comes from the same factory I already have.

public sealed class SqlContainerFixture : IAsyncLifetime
{
    private readonly MsSqlTestcontainer _container = new TestcontainersBuilder<MsSqlTestcontainer>()
        .WithDatabase(new MsSqlTestcontainerConfiguration
        {
            Password = "yourStrong(!)Password123"
        })
        .Build();

    public string ConnectionString => _container.ConnectionString;

    public async Task InitializeAsync()
    {
        await _container.StartAsync();
        await MigrationRunner.RunAsync(ConnectionString);
        await SeedAsync();
    }

    private async Task SeedAsync()
    {
        var users = new UserFactory().Build(20);
        await BulkInsert.UsersAsync(ConnectionString, users);
    }

    public Task DisposeAsync() => _container.DisposeAsync().AsTask();
}

The first run hurts because the MSSQL image has to be pulled and the server needs a moment before it starts accepting connections. In a warmed-up environment, this takes a dozen or so seconds. That is the price I pay for knowing that nobody left anything in this database before me.

The greatest temptation with this approach is to start one container for the entire run and share it between test classes to save those dozen or so seconds. I have done that, and I advise against it if parallel execution is a consideration. The moment two classes run against the same instance at the same time, we return to exactly the first pain from the beginning of the post, only with an extra container along the way. If I do share an instance, each class gets its own database inside it or its own schema, not the same tables.

The container itself has no opinion about what should be in a record. It provides an empty, isolated database. The content still comes from the factory, and these two things do not replace each other.

No database: WireMock.Net 1.4.20 in test code

The fourth layer concerns data that is not in my database at all because it belongs to someone else’s service. In May, I started WireMock as a container in Compose, with mappings in JSON files. I still use that path for an environment that runs alongside the application. Today I am showing the second variant, closer to the topic of this post: the WireMock.Net library version 1.4.20 from August 6, launched directly from .NET test code.

public class CatalogClientTests : IAsyncLifetime
{
    private WireMockServer _catalog;

    public Task InitializeAsync()
    {
        _catalog = WireMockServer.Start();

        _catalog
            .Given(Request.Create()
                .WithPath("/api/products/1001")
                .UsingGet())
            .RespondWith(Response.Create()
                .WithStatusCode(200)
                .WithHeader("Content-Type", "application/json")
                .WithBodyAsJson(new
                {
                    id = 1001,
                    name = "400 ml thermal mug",
                    price = 59.9m,
                    currency = "PLN",
                    inStock = true
                }));

        _catalog
            .Given(Request.Create()
                .WithPath("/api/products/9999")
                .UsingGet())
            .RespondWith(Response.Create()
                .WithStatusCode(503)
                .WithDelay(TimeSpan.FromSeconds(3)));

        return Task.CompletedTask;
    }

    [Fact]
    public async Task Unavailable_catalog_does_not_break_cart()
    {
        var client = new CatalogClient(new Uri(_catalog.Urls[0]));

        var result = await client.GetProductAsync(9999);

        Assert.False(result.IsAvailable);
    }

    public Task DisposeAsync()
    {
        _catalog.Stop();
        return Task.CompletedTask;
    }
}

Three things distinguish this variant from the one in May, and they are why I am including it here.

The mock data is in test code, so it is compiled and refactored with everything else. This is the answer to the third pain from the beginning. In May, I made sure the mappings lived in the repository and went through review like code, but a JSON file is still not compiled with the project, so nobody gets a warning when the stub stops matching the client. A stub in a test method is visible in the same diff as a change to the HTTP client.

WireMockServer.Start() with no arguments takes a free port and returns it in Urls. I do not hardcode the port number because that guarantees a collision when tests run in parallel on one agent. I inject the address into the client, which again requires that address to be configurable rather than embedded in production code.

The error scenario is just as cheap here as the happy path. A 503 with a three-second delay takes five lines, and I have no way to trigger it on demand in the partner’s sandbox.

This still does not guarantee compatibility with the contract, and I still keep a small group of tests outside this suite that hit the real sandbox and check only the response shape.

Isolation versus speed

The four layers differ primarily in how much I pay for isolation. It is worth putting them in order.

The cheapest option is a transaction with a rollback: I open a transaction before the test, do my work, and roll it back. It takes milliseconds and is tempting. However, it has two limitations that I have encountered in practice. The code under test must use my connection, so a scenario where the application responds over HTTP and accesses the database through another connection is out. The second limitation is subtler: the transaction changes the behavior of what I am testing. Code that manages transactions itself or relies on another session seeing data after a commit will behave differently than in production, and I will get a green test for a situation that never occurs.

Respawn sits in the middle. A reset takes a fraction of a second instead of dozens of seconds of migrations, works regardless of who writes to the database and from where, and does not change the code’s behavior. It costs enough for the state between tests to really disappear, so there is nothing to take shortcuts with.

A new container is the most expensive and safest option. A dozen or so seconds of startup in exchange for a guarantee that nobody has been here before.

I have a hypothesis about this that I have not yet had time to measure properly, so I present it as a hypothesis. In CI with parallel jobs, state local to the job is the safest: its own container or its own stub running in the same process as the tests. A shared resource, even one that is reset, becomes the point where jobs get in each other’s way under parallel execution, and the more agents there are, the more often this happens. A global dump restored once a night is the extreme version of this: everyone looks at the same data and nobody knows who changed it last. I reached similar conclusions with Selenium Grid in containers, where a shared browser turned out to be more expensive than a separate node for each job.

One misunderstanding is worth clearing up here. When I wrote about retry policies with Polly in July, I was solving the problem of a temporary read race: something had not materialized yet, so I ask again after a moment. A retry does not fix bad data. If a test only passes on the third attempt because it hits a different database state on that attempt, a retry is not a cure but a painkiller that takes away the only signal I have that state is leaking between tests. The distinction is simple: retries for what is slow, isolation for what is dirty.

Summary: record the layer choice in the test README

This leaves me with a division that I am adopting as the default from today. Domain logic gets a seeded factory with no database. Queries, mapping, and migrations get a real database reset with Respawn, and if I want to be certain that I am starting from nothing, a container started from the test. Everything that belongs to someone else’s service gets an HTTP stub with data versioned alongside the test.

The most important thing, however, is not the decision tree itself but recording it somewhere. In my case, it goes into README.md in the test directory: which layer applies in which project, where the factory lives, what is on the list of tables skipped by Respawn, and why. Without this file, after three months the repository has all four layers at once, each introduced by someone else, and nobody knows which one to choose when writing a new test. A dozen or so sentences in the README cost less than repeating that conversation every sprint.

I leave two things until the end because they are rules for me, not preferences. A production copy is not a test data strategy until anonymization is a separate, maintained project. A language-model-based generator does not replace a seed: it gives me a dozen or so nice records for a person to review, not a repeatable state that supports an assertion.

And as with every previous tool, it is worth checking where this fits into the entire test process. Organized data does not fix the process. It only makes a red test finally mean “the application is broken” rather than “someone before me left something here.”