GitHub Copilot technical preview - first impressions in VS Code

Gray text that was not there before

Since the end of June last year, the same story has been circulating in my bubble: GitHub showed a tool it called an “AI pair programmer”, some people I know got access, and the rest are still on the waiting list. I joined that list in the week of the announcement and, after my share of waiting, received an invitation. So for the past few weeks I have had something in VS Code that was not there before: gray text that appears in front of me as I type and proposes the rest of a line or an entire function.

I am saying this directly at the outset because in February 2022 it is easy to misunderstand. Copilot did not launch this month. The technical preview announcement is from June 29, 2021, and since then it has been the same thing: a limited preview, a waiting list, and free access for those who make it through. There is no launch, no pricing, and no boxed product. There is a tool in testing that some of those interested get access to.

This post is about how I set it up and what came of several weeks of writing code with it every day. It is not about how I am about to stop writing tests. Just as in last year’s summary, where I put language models in the “desk experiment” category, the same applies here: I am describing a tool I use, not a process change I have just introduced.

What Copilot is in February 2022

In the shortest terms: it is an editor extension that suggests code where the cursor is. There is no chat window. You do not ask it questions and receive answers in paragraphs. It takes context from what is at hand, meaning the current file: imports, class names, the signature of the method I have just started, and a comment I wrote one line above. Based on that, it proposes what comes next, which I see as grayed-out text embedded in the editor.

Underneath is an OpenAI model called Codex. GitHub and OpenAI have been talking about it since June, and in August 2021 OpenAI described Codex separately; there is also a paper describing the evaluation of this family of models on programming tasks. For me as a user, however, this is a technical detail. In practice, I do not see the model or its parameters. I see gray text and decide whether I want it.

The preview page has a list of languages in which it works particularly well: Python, JavaScript, TypeScript, Ruby, Java, and Go. C#, which pays my bills, is not at the top of that list, but since the October 2021 update it has received multiline suggestions on par with Java and C++. In short: the marketing demos are not recorded with NUnit, and yet something is happening in my test code. How much is covered later.

It is worth clearing up two misunderstandings that I hear most often. Copilot is not a search engine for Stack Overflow that pastes a result it found. Nor is it a linter or a template-based generator, because a template is deterministic and this is not. It is a model that predicts what comes next in text, except that here the text is code.

Access and price, meaning the waiting list and telemetry

The first step is not technical. The first step is the form at copilot.github.com and waiting. The number of places in the preview is limited, and GitHub lets people in in batches. I cannot say how long the wait is because there is no published rule for it; I waited my share. I am writing about this in the second section rather than a footnote because I have already seen several guides that begin with “install the extension”, while installation without access only gives you a request to sign in and a message that the account does not have the preview enabled.

The price today: the preview is free for people with access. There is no subscription, no invoice, and no card to enter. GitHub says explicitly that this is a testing phase, not a commercial product, and that the business model is yet to be created. That is a plan for the future, not a fact in February, so if someone is planning a team budget for this year, there is nothing to put in the spreadsheet today.

The third point is more important than the previous two, and it determines whether this can be enabled at work at all. Copilot runs in the cloud. To get a suggestion, a fragment of my file leaves the editor for GitHub’s service. There is also telemetry: the service records which suggestions I accept and which I reject so that the model and product can be improved. You accept the telemetry terms when enabling the extension, and there is no “runs locally, nothing leaves” mode.

In the preview documentation, GitHub answers two questions people ask immediately. Will my private code become someone else’s suggestion? The FAQ says no. Who owns the code I accepted? I do, just like any code I entered into my file. I take note of these answers and still follow a simpler rule because it does not depend on someone else’s FAQ: I enable Copilot in my own public scratch space and where I have clear permission to do so. I do not enable it automatically in a client’s repository because that is not a decision for me to make alone.

Setup in VS Code step by step

The rest of this post takes place in VS Code, version 1.64.2 from February 10. I choose VS Code because it is the environment from the announcement and remains the best-supported one. There is also official support for a Neovim plugin (github/copilot.vim), the JetBrains family (IntelliJ, PyCharm, Android Studio), and Codespaces. As of today, there is no extension for Visual Studio, Microsoft’s large IDE; there is an announcement on GitHub’s public roadmap, and it is an announcement, not something I can install today. If someone uses Visual Studio 2022, for now they are left looking at other people’s screenshots.

Step one: a GitHub account with preview access enabled. You can check this in your account settings, in the Copilot section that appears only after an invitation.

Step two: the extension. From the editor palette or the command line:

$ code --install-extension GitHub.copilot

Step three: signing in. On the first run, the extension asks for authorization through OAuth, opens a browser, and comes back with a token. A Copilot icon appears in the status bar. A crossed-out icon means that suggestions are disabled for this file or globally, and this is the same switch I use when recording my screen or running a workshop.

Step four is my own and is not in any GitHub guide: a separate, empty repository for experimenting. A regular directory with one console project and one test file. I did everything described later there, with made-up data. This has two advantages. Nothing real leaves for an external service, and I see suggestions in a clean context, without the influence of ten thousand lines of someone else’s code.

One disappointment is worth mentioning in advance. Copilot in an empty file, without imports and names, is much weaker than in a file that already contains something. When I open a test class in the suite I wrote about during the refactoring to Page Object Model, the suggestions are much more accurate because the model sees the naming conventions and what the neighboring methods look like. Context is this tool’s fuel.

The accept and reject loop

The entire operation comes down to a few keys, and that is probably the best thing about this product.

I write code as usual. When the model has a proposal, gray text appears after the cursor. Tab accepts it in full. Esc rejects it, and the gray text disappears without a trace. If I do not like the proposal but the direction is right, I move through the next alternatives with Alt+] and Alt+[ (Option on macOS). There is also a panel listing alternatives under Ctrl+Enter, which opens a separate tab with several proposals at once. I rarely use it because it breaks my writing rhythm, but for a longer function it can show that the model has two completely different ideas for the same thing.

I provide context in two ways. The first is a signature, meaning I simply start writing a method and wait:

public class OrderTotalTests
{
    [Test]
    public void Total_WithSingleItem_ReturnsItemPrice()
    {

At this point, Copilot most often adds the body of the test in an arrange, act, assert structure consistent with what it sees in the neighboring methods. The test name is an instruction here, so the more descriptive the name, the less it guesses.

The second way is a comment, meaning I write in a sentence what I want:

// build a cart with three items at 19.99 each and calculate the total with a 10 percent discount

After moving to a new line, I get a code proposal. This is where the part that needs to be said out loud begins: a proposal can be good, and it can be made up. Once, the model proposed an ApplyDiscount method here that does not exist in my project because we named it differently. The code looked credible and did not compile.

Hence the one rule I follow without exception. Every accepted suggestion is someone else’s diff and goes through the same review as a diff from a person. I read it before pressing Tab, and I read it again before making a commit. Tab does not mean “I agree”, only “paste this into my file so I can inspect it”.

What works and what is garbage

I will start with what works well, because it does.

Boilerplate. A constructor assigning fields, mapping a DTO to a model, a set of using directives, building a test object with several fields, another case in a TestCase differing by one value. Wherever the code is repetitive and predictable, the model gets it right and saves me real minutes. This is easiest to see when one example of a given pattern already exists in the file. Then the second and third are created almost by themselves.

Names and small completions. Copilot is good at guessing what to name a variable and how to finish a line that has been started. I treat it as better code completion, not as a new category of tool, and in this mode it works very decently.

Now for the list of things that ruin the day if they are not watched.

Code that does not compile. The method already mentioned that does not exist. An overload with a different number of arguments. A return value of a type different from the declared one. This is apparent immediately and is the least dangerous item on the entire list because the compiler complains.

Old APIs. The model has seen a lot of code, and some of it is years old. I get suggestions with patterns no longer used in .NET 6, or with synchronous calls where my entire path is asynchronous. This compiles, so it has to be caught by eye.

Data that looks real. When generating sample records, it can produce email addresses in realistic-looking domains and numbers that look like personal data. I do not put such things in a repository. In fixtures, I stick to documentation domains, exactly as I did in the data generation experiment.

Security. In August 2021, NYU researchers published a paper in which they ran Copilot suggestions through a set of scenarios built around common vulnerability classes. Many of the generated programs contained a vulnerability: no input validation, incorrect handling of external data, weak use of cryptography. This is not an accusation against the tool, only a description of what it is. The model proposes code similar to code it has seen, and code it has seen can have holes.

And one thing worth knowing in advance. On June 30, 2021, GitHub published its own study on recitation, meaning how often Copilot outputs a fragment that repeats existing code verbatim. It found this in around 0.1 percent of cases, although the distribution is not uniform: recitation happens more often when the file is empty and the model has nothing to latch onto, and it almost never happens in the middle of real code with context. The sentence about the filter is important to me. In this study, GitHub says that a mechanism for detecting such repetitions is not yet integrated into the technical preview, and in this year’s documentation I still read that the filter is being developed. In other words: today this is my responsibility, not a product feature.

The last point from the FAQ, which I repeat at every meeting: Copilot does not run or test the code it proposes. It has no idea whether it works. It is a machine for writing probable text, not for checking whether that text is true.

This is not the same as Completions

In June, I described an experiment with generating test data through the Completions API, and I see people putting both things in the same “language models” bucket. It is worth separating them because they are two different tools for two different tasks.

That experiment was an HTTP call. I sent a POST to an endpoint with davinci in the address, built a few-shot prompt with several example JSON objects, set max_tokens, temperature, and stop, received text in response, validated it in Python, and put the approved result in a file in the repository. I controlled every parameter, paid for tokens, and decided myself when the call happened. The result was data, not code. Incidentally, the waiting list for that API was removed in November, so in that case the barrier to entry disappeared. It remained for Copilot.

Copilot is on the other side. There is no prompt that I write and version myself because my open file is the prompt. There are no parameters to set. There is no JSON response that I can validate against a schema because the response is text inserted directly into the editor. There is no Completion.create in my code and no API key in an environment variable. There is gray text and two keys.

The practical consequence is that these two tools have completely different places in the process. The result from Completions is an artifact: I save it, version it, someone reviews it in a pull request, and a test reads it from disk. The result from Copilot is not an artifact, but the way my code was created. Once accepted, it is indistinguishable from what I typed with my fingers, which is precisely why review before Tab is so important. There is no second place here where an error will be stopped.

Summary

After several weeks, I have a simple conclusion. This is a tool for writing code faster when that code goes through review anyway. It is not a source of truth about whether code works, and it does not replace any step in my process. The compiler, tests, code review, and a second person looking at it all remain exactly where they were.

In practice, I gain from repetitive boilerplate and from completing lines in a file that already has context. In practice, I lose time when I believe the gray text without reading it because debugging a made-up method takes longer than writing it myself. My balance after several weeks is positive, but it is not a gain that changes the way the team works. It is a gain that changes one person’s writing speed.

I will repeat three things at the end because they are confused most often. Copilot in February 2022 is a technical preview behind a waiting list, announced in June 2021, and this is not a launch this month. It is free for people with access, while pricing is a plan for the future, not a fact. And it provides suggestions in the editor, not a conversation: there is no window in which I ask the tool anything.

I am left, however, with the question that has followed me since my first day with this extension: can it sensibly speed up test scaffolding? Not the content of the assertion, because I have to know that myself, but all the repetitive surroundings - the test class, parameterized cases, configuration. I want to test this properly on one specific suite and describe it separately.

Before I start rearranging anything, however, I return to the question I ask with every new tool: where does this fit into the whole testing process? Writing code faster does not fix a process. At most, it gets me more quickly to the point where I can see what does not work in it.