All posts

Copilot for NUnit and xUnit test boilerplate
Copilot for NUnit and xUnit test boilerplate

Polski

Copilot for NUnit and xUnit test boilerplate

March 2022: Copilot technical preview drafts NUnit/xUnit methods and page objects. Ghost text only; review assertions and locators. Not chat.

Introduction

In February, I described Copilot’s technical preview here: access from the waitlist, the VS Code extension, gray text appearing in front of the cursor, Tab, and Esc. That post was about the tool. I checked that it worked and put the subject aside for a month. March is the first month in which I have kept the extension enabled all the time in a project I actually maintain, so I can ask a narrower and more useful question.

The question is this: does ghost text shorten the boring part of writing a test, or does it only produce green methods without assertions? The boring part is real and measurable. It is the method name, navigation to an address, three braces, a using, a repeated attribute, another [InlineData], an empty component class with five locators. Copilot’s landing page advertises exactly that: “tests without the toil” and “autofill for repetitive code”. In March 2022, I am checking how much of that promise remains when the stack is C#, not JavaScript from GitHub.

The application under test is still the same public RealWorld-style demo I used when refactoring page objects: login, a header with links for a logged-in user, and an article list. No client code. The solution has two test projects because that is how it developed historically: the UI uses NUnit, while the helper layer and data preparation logic use xUnit. Both target net6.0 and have pinned versions.

<PackageReference Include="NUnit" Version="3.13.2" />
<PackageReference Include="NUnit3TestAdapter" Version="4.2.1" />
<PackageReference Include="Selenium.WebDriver" Version="4.1.0" />
<PackageReference Include="xunit" Version="2.4.1" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.3" />

The extension is GitHub.copilot in VS Code 1.65.2. I did not install anything else, and I do not need anything outside the editor here.

Context, not chat

The first thing to get straight before complaining about the quality of suggestions is that there is nowhere here to ask for anything. There is no text field, no answer window, no “write tests for this class for me”. There is an editor, a cursor, and gray text. The entire input channel is the file.

In practice, I have three levers, and all of them are in the code.

The first is a comment above the signature. A regular // or an XML comment, one sentence in Polish or English, describing behavior rather than implementation. The second is the method name. If tests in the project are named as sentences, the name is half of the prompt and the model adds the body beneath it. The third, and least obvious, is neighboring tabs. The preview FAQ says directly that the extension sends the contents of the edited file and, in some cases, content from neighboring files as well. This is not a theory. I returned to the demo below twice: once with only the test file open, and once with LoginForm.cs open next to it. The suggestions were different, and I will show how in a moment.

It is worth comparing this with what I did with the Completions API in June. There, I built a prompt with several few-shot examples, set the temperature and token limit, sent HTTP, and received JSON that I then validated against a schema. I had full control over the input and full responsibility for joining it to the project. Here, it is the opposite: zero control over the prompt, zero parameters, but the output lands immediately in the right file, in the right class, with the right types. These are two different tools for two different problems. That one generated data; this one generates a skeleton.

Demo 1: NUnit method

The test class has existed since the August refactor. It has [FixtureLifeCycle(LifeCycle.InstancePerTestCase)], a driver in an instance field created in the constructor, and Dispose, which closes it. Next to it is one finished test for a successful login. I add a comment and start typing the attribute.

// logging in with an invalid password shows a validation message above the form
[Test]

The first suggestion, with only the test file open:

[Test]
public void Should_ShowMessage_WhenPasswordIsInvalid()
{
    _driver.Navigate().GoToUrl($"{BaseUrl}/login");

    var login = new LoginPage(_driver);
    login.Form.Fill("test@test.com", "wrong-password");
    login.Form.Submit();
    Thread.Sleep(2000);

    Assert.IsTrue(login.Form.ErrorMessages().Count > 0);
}

It looks sensible for the first three seconds and has four separate problems.

Fill is private in my LoginForm, and Submit does not exist at all. The model suggested an API that sounds like mine but is not there, so this does not compile. This is actually the best kind of error because the compiler catches it. The ones that compile are worse.

Thread.Sleep(2000) is exactly the line I removed from the entire suite in August. The model does not know that it is absent from this repository because its training data is full of it.

Assert.IsTrue(...) is NUnit’s old assertion model. When it fails, the report will show Expected: True, But was: False, and that is all. The constraint model, meaning Assert.That, shows the actual value in that situation.

Most importantly, the assertion checks that there are more than zero messages. The test will also pass if the application displays a message with completely different content, for example one about the server being unavailable. A green result would then mean nothing.

Now the same thing with LoginForm.cs open next to it. The suggestion changes in a way that explains the previous section about context well:

[Test]
public void InvalidPasswordShowsValidationMessage()
{
    _driver.Navigate().GoToUrl($"{BaseUrl}/login");

    var form = new LoginPage(_driver).Form
        .SubmitExpectingFailure("test@test.com", "wrong-password");

    Assert.That(form.ErrorMessages(), Is.Not.Empty);
}

The model saw SubmitExpectingSuccess and SubmitExpectingFailure and started using them. Thread.Sleep disappeared because the wait sits in the component and there is no trace of it in the visible code. The naming convention from another repository disappeared. The weak assertion remained, and I correct it myself to Is.EqualTo with the specific message text.

The result is a test that looks practically the same as the one I wrote by hand in August, and that is exactly the point. Copilot did not invent a better pattern for me. It reproduced the one that was already in neighboring files and saved me some typing. Out of curiosity, I measured this across five more tests: the saving is around one third of keystrokes and zero percent of decisions. The name, navigation, braces, and component call write themselves. I still have to know what should be in the assertion.

Demo 2: xUnit draft

The second project has no browser. In it, I test a helper that turns an article title into a slug because the API tests need to predict the address where the created article will be found. The rule is simple: lowercase letters, spaces changed to hyphens, special characters removed. I write a comment and [Theory].

The first observation is that until I typed [Theory] myself, the model suggested three copied [Fact] methods with a different constant in each one. That works, but it falls apart with the fourth case. I had to start the parameterization myself.

[Theory]
[InlineData("Hello World", "hello-world")]
[InlineData("  Trailing spaces  ", "trailing-spaces")]
[InlineData("C# and .NET 6", "c-and-net-6")]
public void Slugify_ReturnsExpectedUrlShape(string title, string expected)
{
    Assert.Equal(expected, Slug.From(title));
}

This is a good skeleton, and I accept it almost in full. But immediately afterward, the extension starts adding more rows because [InlineData] is a perfect shape for a model predicting the next line. Among other things, I received a suggestion with an email address that looked real, containing a first and last name and using the domain of a popular email provider. I delete things like that reflexively. GitHub itself published a study in 2021 about how often the model recites fragments of training data, and with personal data this is not a curiosity but a category to reject without a second thought. There is also a completely mundane concern: I do not want something in the repository that someone will take for a real user’s address during the first audit.

The second thing to check in this theory is less striking but more important. The three generated cases go through the same branch of the code. They all contain ASCII characters. Meanwhile, the only branch that can actually break in my implementation is Polish diacritics, and the model did not suggest such an [InlineData] even once until I typed half of it myself. A theory with three cases looks like a test that is three times better than a [Fact], but it can be one test run three times.

This is the same problem I described with test data management strategies. Data invented on the spot confirms what we already know. Data chosen for a specific branch or produced with a fixed seed shows something new. Copilot is excellent at generating the first category and does not know the second at all because it has no idea what my implementation of Slug.From looks like beyond what it sees in the file.

Demo 3: page object, not 50 FindElement calls

The third scenario is the one I cared about most. I create a new component file, write a one-sentence comment in the class, and watch what gets added.

public sealed class LoginForm
{
    private readonly IWebDriver _driver;

    public LoginForm(IWebDriver driver) => _driver = driver;

    public void TypeEmail(string email) =>
        _driver.FindElement(By.Id("email")).SendKeys(email);

    public void TypePassword(string password) =>
        _driver.FindElement(By.Id("password")).SendKeys(password);

    public void ClickLogin() =>
        _driver.FindElement(By.XPath("//*[@id='root']/div/div/form/fieldset/button")).Click();

    public bool IsErrorDisplayed() =>
        _driver.FindElement(By.ClassName("error-messages")).Displayed;
}

It compiles and is not useful for anything. Point by point, according to the rules I set in August.

The locators are invented. By.Id("email") sounds credible, but this demo has no such identifier. The absolute XPath in ClickLogin comes from some other DOM tree and will break with the first layout change. This is the most treacherous category because the code passes review by eye and only fails during a run.

The API is a collection of setters. A test would have to call three methods in sequence and know the order itself, so the mechanics return to the test method I was trying to keep them out of.

IsErrorDisplayed is an assertion smuggled into the page object. It returns a bool, so the test will end with Assert.That(form.IsErrorDisplayed(), Is.True), and when it fails I will see Expected: True again. The component should return data, not a verdict.

There is no waiting at all. There is no By declared as static readonly. On the other hand, when I left an empty file with no context and only the comment “home page page object”, I got a class with over a dozen fields and all of the application’s selectors in one place, which is exactly the fat HomePage I spent two weeks breaking apart. Not because the model is stupid, but because that shape dominates public code.

Reversing the order was the only thing that worked. First, I type the locator block myself, then I start the method:

private static readonly By Root = By.CssSelector("form");
private static readonly By EmailInput = By.CssSelector("input[type=email]");
private static readonly By PasswordInput = By.CssSelector("input[type=password]");
private static readonly By SubmitButton = By.CssSelector("button[type=submit]");
private static readonly By ErrorItem = By.CssSelector(".error-messages li");

From that point, ghost text stops inventing selectors and starts using mine. It adds Container, adds searching inside the container, and adds WebDriverWait with a lambda when it sees that such a wait is present in a neighboring component. The conclusion is simple and repeats across all three demos: Copilot copies the local convention if the convention is visible in the file. If it is not, it copies the convention of the internet.

What I reject immediately

After two weeks, I have a list of things I do not even finish reading. I press Esc and write the code myself.

  • A test without an assertion or with an assertion that “no exception was thrown”.
  • An absolute XPath such as //*[@id='root']/div/div/form. Copied from someone else’s DOM and invalid in mine.
  • An email address, password, or number that looks like a real person’s data.
  • A test that already exists three methods earlier under a different name. This happens often because the neighborhood is the model’s strongest clue.
  • A call to a method that does not exist in the project. The compiler will catch it, but it costs a minute and breaks my focus.
  • public static IWebDriver in any form.
  • Thread.Sleep in any form.

It is also worth noting what the extension does not do in this preview: it does not say where a particular suggestion came from, and there is no panel where I could ask about it. All verification is on my side, and it is code verification, not a conversation.

Running it is part of the review

The simplest rule from this month is: I do not commit a test I have not run. It sounds trivial until a test writes itself in two seconds and looks correct.

dotnet test --filter "FullyQualifiedName~LoginTests"

The model that suggested a test method to me did not run my application even once. It does not know the exact text the form shows for an invalid password, how long the redirect takes, or whether .error-messages is a child of <form> or its sibling. It only knows what code like this usually looks like. A green result from such a test is worth as much as the verification I perform on it myself.

So I do two runs instead of one. The first confirms that the test passes. The second confirms that it can turn red: I temporarily change the expected message text and check whether the test actually fails and whether NUnit’s error message is readable. If the test is still green after replacing the expected value, it does not check what I think it checks. This second run caught three generated assertions for me in March.

Copilot does not replace anything below the code layer either. When I started writing an integration test where I start the database with Testcontainers, ghost text very smoothly wrote a container configuration using an API that did not match the package version I had pinned. The extension does not know which image I have in the registry, how much memory the CI machine has, or whether the port is free. The test infrastructure remains one hundred percent mine, and the farther I move from pure code, the less benefit I get from this tool.

And the most obvious thing, which is easiest to forget with a tool that provides instant gratification: review before sending something to CI is not a formality, but part of the test process. The writing speed has just increased, so controlling what enters the suite becomes more important, not less.

Summary

After two weeks of daily work with this extension, the answer to the question from the introduction has two parts. Boilerplate, yes; responsibility for the suite, no.

Yes, ghost text really does shorten the boring part. Method names, attributes, navigation, another [InlineData], a component skeleton, a using at the top of the file. One third less typing is not a revolution, but it is noticeable and does not require me to change tools or process.

No, it does not remove a single decision from me. The assertion, locator, choice of data for a branch, and fixture lifecycle are four places where the suggestion was systematically plausible instead of correct. The three rules that remain are short: write the convention before asking for code, read the assertion and locator character by character, run it locally, and check whether it can turn red.

Copilot is still in technical preview, still behind a waitlist, still only in the editor. It will leave this preview someday, and then its price and availability will change, not the nature of the tool. It suggests the next line based on what is already in the file, and as long as that remains true, the review habit will be exactly the same as it is today.