xUnit and NUnit on .NET 6 - Fact, fixtures, collections

Introduction

In April, I started MSSQL with Testcontainers in a project created with dotnet new xunit. In August, the page object refactor used NUnit and Selenium. In October, I wrote about parallel tests on CI. In the year in review I closed 2021 on an isolation thesis, and the blog itself shows that I spent the year switching between xUnit and NUnit without writing down any rules. January is a good time to catch up because the SDK in my projects is already on .NET 6 and templates created from scratch target net6.0.

This post is not about which framework is better. It is about one question that has to be answered consciously in both cases: where state lives. A test class has fields, a fixture has a database connection, and a container has a port. If I do not know when this is created and when it disappears, I also do not know whether two tests can run side by side. Everything else, namely attributes, filters, and runner configuration, follows from this one decision.

I am not migrating Selenium here or cutting pipeline time. The demo is synthetic: two test projects for the same application, one using xUnit, the other NUnit, both with one class that needs a database and one that needs nothing. I have pinned the versions and intend to stick to them throughout the post:

<PropertyGroup>
  <TargetFramework>net6.0</TargetFramework>
  <ImplicitUsings>enable</ImplicitUsings>
  <Nullable>enable</Nullable>
</PropertyGroup>

<ItemGroup>
  <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.0.0" />
  <PackageReference Include="xunit" Version="2.4.1" />
  <PackageReference Include="xunit.runner.visualstudio" Version="2.4.3" />
</ItemGroup>

On the NUnit side, it is NUnit version 3.13.2 from April 27 last year plus NUnit3TestAdapter 3.17.0, which is exactly the same set used by the August refactor. I am changing the target framework, not the test library.

Two instance models

The most important difference between these frameworks is not the name of an attribute, but the test class lifecycle.

xUnit creates a new class instance for every [Fact]. The constructor is the equivalent of SetUp, and Dispose is the equivalent of TearDown. There is no attribute to enable because this is how the framework has always worked, and it cannot be disabled:

public class CartTests : IDisposable
{
    private readonly List<string> _events = new();

    public CartTests() => _events.Add("ctor");

    [Fact]
    public void FirstTestSeesOneEvent() => Assert.Single(_events);

    [Fact]
    public void SecondTestAlsoSeesOneEvent() => Assert.Single(_events);

    public void Dispose() => _events.Clear();
}

Both tests pass and would pass even without Dispose, because _events from the first test does not survive long enough to reach the second one at all. It is the same list, but not the same object.

By default, NUnit does the opposite: one class instance handles all tests in a fixture. A direct rewrite of the example above looks like this, and the second test fails:

[TestFixture]
public class CartTests
{
    private readonly List<string> _events = new();

    [SetUp]
    public void SetUp() => _events.Add("setup");

    [Test]
    public void FirstTest() => Assert.That(_events, Has.Count.EqualTo(1));

    [Test]
    public void SecondTest() => Assert.That(_events, Has.Count.EqualTo(1));
}

The second test sees two entries because [SetUp] ran twice on the same object. I usually do not detect this so quickly because in real code the field is overwritten, not appended to, and for a year nobody notices that state is leaking. It is noticed only by the first run in which the test order differs from usual.

Since NUnit 3.13, released on January 7 last year, there has been an attribute for this, and it is the one I described when writing about page objects:

[TestFixture]
[FixtureLifeCycle(LifeCycle.InstancePerTestCase)]
public class CartTests : IDisposable

With this attribute, NUnit behaves like xUnit: a new instance for every test case, the constructor before the test, and Dispose after the test. I prefer to set this once for the entire assembly with [assembly: FixtureLifeCycle(LifeCycle.InstancePerTestCase)] rather than remember the attribute in every new class.

There is one more consequence that is easy to forget when rewriting tests. xUnit has no equivalent of [SetUp], and that is intentional: because the constructor runs before every test, another mechanism would be the same thing under a different name. When I move a class from NUnit, the method marked [SetUp] has to end up in the constructor or be called manually, while [TearDown] goes into Dispose. Automation will not do this, and there is no compiler warning because an orphaned SetUp method is simply an ordinary method that nobody calls. It is worth remembering because the test then passes for reasons nobody has verified.

And this is the point: in xUnit, instance isolation is the default, while sharing context requires an explicit declaration. In NUnit, the reverse is true: sharing is the default, while isolation requires an attribute. Both models can be taken to the same place, but the directions from which they arrive are opposite, which is why transferring habits between frameworks hurts.

Attributes that actually organize the suite

Once the lifecycle is clear, what remains is the layer we usually think of as “syntax.” In practice, these attributes determine two things: how many tests I have instead of one loop, and whether they can be filtered on CI.

Parameterized tests in xUnit are [Theory] with [InlineData] for constant values and [MemberData] for everything that has to be calculated:

[Theory]
[InlineData("", false)]
[InlineData("abc", false)]
[InlineData("user@example.com", true)]
public void AddressValidation(string input, bool expected) =>
    Assert.Equal(expected, EmailValidator.IsValid(input));

public static IEnumerable<object[]> Discounts() =>
    new[] { new object[] { 100m, 0.1m, 90m }, new object[] { 50m, 0m, 50m } };

[Theory]
[MemberData(nameof(Discounts))]
public void DiscountCalculation(decimal price, decimal discount, decimal expected) =>
    Assert.Equal(expected, Pricing.Apply(price, discount));

In NUnit, the same two cases are [TestCase] and [TestCaseSource]. There is one practical difference, and it is worth knowing before a migration: NUnit can return TestCaseData with a name and category assigned to an individual case, while xUnit works with object[] arrays and builds the name from argument values. When input data is long, an xUnit report becomes less readable, and that is an argument for [MemberData] returning types with a proper ToString(), not raw strings.

Categorization is [Trait] in xUnit and [Category] in NUnit:

[Fact]
[Trait("Category", "Integration")]
public async Task SavingAnOrderHitsTheDatabase() { }

[Test]
[Category("Integration")]
public async Task SavingAnOrderHitsTheDatabase() { }

Both ultimately come down to dotnet test --filter: --filter "Category=Integration" for NUnit and the same expression for a trait with the Category key in xUnit. I keep exactly three values: Unit, Integration, Ui. I have seen projects with fifteen categories, and it ends with nobody knowing which one to enter, so new tests do not get any. A category that is only on half the tests is not suitable for filtering because I do not know what I will get after excluding it.

It is also worth deciding whether a category is a property of a test or a property of a project. My answer today is: a property of a project. I keep unit and integration tests in separate csproj projects because then the division is enforced by references, not by the author’s memory. The attribute remains as a second sieve inside the integration project, for example to exclude a handful of tests requiring something that is not available on the agent.

xUnit: fixtures and collections

If a class instance lives as long as one test, where do I put a database that takes ten seconds to start? Not in the test class constructor, because I will pay ten seconds for every [Fact]. Fixtures are for this purpose.

IClassFixture<T> is one object for the entire test class. xUnit creates it once, injects it through the constructor of every instance, and cleans it up after the last test in the class:

public sealed class DatabaseFixture : IAsyncLifetime
{
    public string ConnectionString { get; private set; } = string.Empty;

    public async Task InitializeAsync()
    {
        ConnectionString = await TestDatabase.StartAsync();
    }

    public async Task DisposeAsync() => await TestDatabase.StopAsync();
}

IAsyncLifetime matters here because starting the database is asynchronous, and a constructor cannot be async. xUnit calls InitializeAsync before the first test and DisposeAsync after the last one.

When several classes need to share the same database, a collection is required. The definition consists of three elements, and all three must be in the same assembly as the tests:

[CollectionDefinition("database")]
public sealed class DatabaseCollection : ICollectionFixture<DatabaseFixture>
{
}

[Collection("database")]
public class OrderRepositoryTests
{
    private readonly DatabaseFixture _db;

    public OrderRepositoryTests(DatabaseFixture db) => _db = db;

    [Fact]
    public async Task SavingAnOrderIncreasesRowCount()
    {
        await using var repo = new OrderRepository(_db.ConnectionString);
        var before = await repo.CountAsync();

        await repo.AddAsync(new Order("PL-1", 100m));

        Assert.Equal(before + 1, await repo.CountAsync());
    }
}

The DatabaseCollection class is empty, and that is how it should be. It is only the place where the collection name meets the fixture type. [CollectionDefinition] without ICollectionFixture also makes sense because it groups classes, but does not share anything between them.

The most important consequence is that a collection is the unit of parallelism in xUnit. Tests inside one collection run sequentially, while collections run in parallel relative to one another. By default, each class is a separate collection, so putting [Collection("database")] on four classes not only gives them a shared database, but also makes them run sequentially relative to one another. This is sometimes exactly what I want because four classes writing to one database would interfere with one another anyway. However, I need to know that I pay for it with time and do not get sharing for free.

I reduce the choice between IClassFixture and a collection to one question: is the cost of creating this object greater than the cost of running the classes that use it sequentially? A database container is worth a collection. A configuration object built once or a generated token is not because it is cheap to recreate, and losing parallelism is not worth the price.

An alternative that I see and do not recommend is static with lazy initialization. It works until I need to clean up after myself because a static field has no moment at which the runner could call Dispose on it. The container then remains on the machine until Docker is restarted.

NUnit: FixtureLifeCycle and Parallelizable

In NUnit, the equivalent of a collection fixture looks different because the framework does not inject anything into the constructor. [SetUpFixture] provides a shared context for multiple classes in one namespace:

[SetUpFixture]
public class DatabaseSetup
{
    public static string ConnectionString { get; private set; } = string.Empty;

    [OneTimeSetUp]
    public async Task StartDatabase() => ConnectionString = await TestDatabase.StartAsync();

    [OneTimeTearDown]
    public async Task StopDatabase() => await TestDatabase.StopAsync();
}

A class marked with [SetUpFixture] runs once for all fixtures in its namespace and subnamespaces. This has one disadvantage compared with an xUnit collection: the connection is accessed through a static field, so the test class signature does not show that it needs anything. When I forget [Collection] in xUnit, I get a clear message at startup that the constructor parameter has no matching fixture. In NUnit, a forgotten [SetUpFixture] gives an empty string halfway through the test and a connection error three methods later.

The second thing to remember with InstancePerTestCase is that [OneTimeSetUp] and [OneTimeTearDown] in the test class itself must then be static because there is no longer one instance on which NUnit could call them. The compiler will not catch this; I only get an error during the run.

There is also a difference in when a shared resource disappears. An xUnit collection cleans up after the last test that used it, so the container lives exactly as long as that group of classes runs. [SetUpFixture] lives until the end of the run in a given namespace, regardless of how many tests were actually run after filtering. This makes no difference for a full run, but it does for dotnet test --filter on one class because I pay for starting a database I barely use.

The third difference is parallelism itself. NUnit does not enable it by default, and it is enabled with the [Parallelizable] attribute at the assembly, class, or method level. I covered this in the October post about CI together with the YAML, and I will not repeat it. I will only point out the order: the attribute makes sense only when the lifecycle is already set to an instance per test and I know what is inside [SetUpFixture]. The reverse order, first [Parallelizable], then looking for the reason tests are red, costs several days.

Contrast with the 2021 posts

Three things from last year look different today, and I want to state them directly instead of pretending that I wrote them correctly from the start.

In the April post about Testcontainers, the container started inside [Fact] in an await using block. For a demo, this is the right form because the entire mechanism is visible on one screen of code. In a suite with twenty integration tests, the same approach means twenty container starts and twenty times the same amount of time. Today, such a container goes into IAsyncLifetime on a fixture and is shared through a collection. The library is still DotNet.Testcontainers from the 1.5 line.

In August, when refactoring page objects, I showed FixtureLifeCycle with IWebDriver. It was the same attribute in a different context: the driver was supposed to stop being static. The principle is identical, only the object is more expensive.

One question remains that none of these attributes solves: what to put in the shared database. A fixture says when the database starts, not what data it should contain. I wrote about this when discussing test data management strategies, and with a shared fixture it becomes more acute because a test that assumes an empty table stops working when a neighbor from the same collection adds something to it.

.NET 6 pitfalls

The new SDK does not change runner semantics, and that is probably the most important sentence in this section. dotnet new xunit on SDK 6 produces a net6.0 project with C# 10, but the test class lifecycle is the same as it was in 2018. There is no flag that turns xUnit into NUnit or vice versa.

A few small things I encountered when moving projects:

• ImplicitUsings does not include test library namespaces. System.Collections.Generic disappears from the top of the file, while using Xunit; and using NUnit.Framework; remain. They can be added manually with <Using Include="NUnit.Framework" /> in the csproj, and then test files really do become shorter.
• A static HttpClient is a recommendation in production code and a trap in tests. The object itself is thread-safe, but DefaultRequestHeaders is not: a test sets the authorization header and leaves it for the next one.
• A database without a reset between tests behaves correctly with a sequential run and randomly with a parallel one. Test order is not a contract of either framework.
• A fixed port in a fixture blocks two test projects running at the same time on one machine. The container should get a port from Docker and expose it through the fixture.

And one thing I have to repeat because the temptation returns every few months. Retrying is not isolation. I described Polly policies in July and still use them, but they address a temporary failure of an external service, not a state leak between tests. A retry applied to a test that lost a race for a row in the database turns the only signal of the problem into a green result on the second attempt.

Summary

For me, choosing a runner is no longer a matter of taste, but it has not become a matter of ranking either.

• I choose xUnit 2.4.1 when I want instance isolation by default and explicit sharing. A fixture injected through the constructor directly states what the class needs, and a collection is both a unit of sharing and a unit of parallelism.
• NUnit 3.13.2 stays where the team already has hundreds of [Test] and [TestCase] tests. Then the first thing I do is [assembly: FixtureLifeCycle(LifeCycle.InstancePerTestCase)], and only the second is a discussion about [Parallelizable].
• Organizing fixtures is a prerequisite for parallelism, not its consequence. Tests can be isolated without parallelism, but not the other way around.

What is deliberately missing here: pipeline configuration and the number of threads. This is a separate topic from three months ago, and mixing it with code organization ends with me optimizing run time before I know whether the result is reliable. Nor is there an answer here to the question of whether these tests examine the right things at all. An organized fixture lifecycle does not replace a discussion about the entire testing process; it only means I have something to work with during that discussion.