GA today
Today, 21 June 2022, GitHub Copilot is leaving technical preview. The preview was announced on 29 June 2021, so the waitlist gate remained in place for almost exactly a year. As of today, there is no list. There is a price.
The GitHub Changelog puts it simply: individual access costs $10 per month or $100 per year. There is also a 60-day free trial. Verified students and maintainers of popular open source projects get Copilot for free. Anyone who was in the technical preview has free access until 22 August 2022, after which they either start a subscription or the extension is disabled.
Today I have seen reports online in which people from the preview immediately get a trial prompt in the editor, and others in which nothing changes. I am not going to adjudicate screenshots. I am sticking to what the Changelog says: free access for preview users lasts until 22 August, the trial is 60 days, and the price is 10 or 100 dollars. The rest is an implementation detail that will sort itself out within a week.
I am writing this post because I feel that one thing is getting lost in coverage of today’s launch. For me, as someone who has used the preview for several months, the access gate and billing change today. The way I work does not. It is still gray text in the editor, still Tab and Esc, and still no place where I could ask it anything. If someone was waiting for the end of the preview to add a new category of features, they did not get one today.
GitHub also repeats its own metric in the announcement: in files with Copilot enabled, about 40 percent of the code is said to be written by the model, with the claim limited to popular languages. This is GitHub’s claim from launch day, not my measurement. I measured something completely different and narrower in my own work, which I wrote about when discussing test scaffolds: savings of around one third of keystrokes and zero percent of the decisions. Those two figures are not the same figure and should not be compared.
What I am buying, and what I am not
I am buying an individual subscription. That is all there is to buy today, and it is worth saying directly because I am already seeing questions such as “how much does it cost for a team of fifteen people?”
There is no business plan today. GitHub says in its announcement that a version for companies will arrive later this year. That is an announcement, not a price-list item, so it cannot be added to a budget today and cannot be purchased today. There is also no license assigned to an organization and no central seat management. If five people on a team are to have Copilot, today that means five separate individual subscriptions on five personal accounts.
For me, working on B2B contracts, this has a specific consequence. The subscription is attached to my GitHub account, not to the organization where I happen to be working. I therefore pay for it myself and treat it like any other tool in my toolbox, exactly like an editor license or hosting. This simplifies one thing and complicates another. It simplifies the expense because I do not have to ask anyone to approve it. It complicates the decision about where I enable the extension because having a subscription is not the same as having permission to send fragments of someone else’s repository to an external service. That rule does not change in the slightest today: I have Copilot enabled where I have clear permission to use it and disabled everywhere else. A product’s move from preview to GA is not the code owner’s permission.
I treat the 60-day trial for what it is: a window in which to calculate whether $10 per month is a good price for this particular tool in my particular stack. I already know the answer roughly because I have used the preview for months, but I documented February and March, so I have something to compare it with.
The IDE on GA day
Today’s environment looks the same as yesterday’s, only without the waitlist.
I work in VS Code every day. The current line is 1.68, specifically 1.68.1 from 14 June, and that is the version in which I am checking all of this. The extension is still GitHub.copilot, installed the same way as in February, with one difference: after signing in, there is no longer a message saying that I do not have preview access, but a path to a subscription instead.
There are several options besides VS Code, and all of them predate today. The Visual Studio 2022 extension was released on 29 March, so anyone using Microsoft’s large IDE has had it since spring. There is a Neovim plugin and there is the JetBrains family. The choice of editor is not a strategic decision here because the product does the same thing in each of them.
And the most important sentence in this section, because I see expectations of something bigger returning with GA: this is still a code completion extension, not a conversation panel. There is no chat window. There is no field where I type a question and receive an answer in paragraphs. There is no “explain this class to me” or “write tests for this file” as a command issued to the tool. There is a cursor, there is file context, and there is gray text in front of the cursor.
A day with ghost text
Since the way of working has not changed, describing what the day actually looks like makes more sense than describing the installation. I have two keys for this and a fairly clear division.
I press Tab when a suggestion repeats something that is already in the project. Specifically: a constructor that assigns fields, mapping a DTO to a model, another parameterized case that differs by one value, a set of using directives at the top of a file, or the skeleton of a method whose signature I wrote myself. The common denominator is always the same. A suggestion is good precisely when the pattern is already visible in the file or in a neighboring tab. Copilot does not invent a better convention for me. It reproduces the one it sees. If it cannot see one, it reproduces the conventions of the internet, which in test code means Thread.Sleep, absolute XPath expressions, and assertions that say “something is there.”
I press Esc without reading to the end in four cases, and this list has not changed since March.
- An assertion. This is the only place in a test that actually makes a statement, and the model has no way of knowing what that statement should be. A suggestion such as “the message list is not empty” is probable and worthless because it also passes when the application displays a completely different message.
- A locator. A selector generated from memory looks credible and comes from someone else’s DOM tree. This is the worst category because it passes an eyeball review and fails only when run. The rules I established when refactoring page objects apply just as much when the code wrote itself.
- Anything that looks like a secret or real personal data. An email address in the domain of a popular email provider, a phone number, or anything that someone might take for user data during an audit.
- Code I do not understand. If I have to spend more than a moment working out what a suggestion actually does, writing it myself is faster than deciphering it.
The change the last quarter has brought is not in the tool, but in my pace. I write code faster, which means I reach the point where a decision has to be made sooner. Reviewing the diff before the commit remains entirely my responsibility and therefore becomes more important, not less. I do not commit a test that I have not run, and I make a second pass in which I check whether the test can fail at all. This is the same rule I have followed for years, only applied more often.
Filter and recitation
One thing in the product changes today besides the prices, and it is worth noting because it concerns what ends up in the repository.
Since June, the settings have included an optional filter that blocks suggestions matching public code in blocks of roughly one hundred and fifty characters or more. This is a switch, not a tool for checking the provenance of code. It does not show which repository something came from, and it does not show the license. It either blocks or does not block.
The context for this option comes from last year. On 30 June 2021, GitHub published its own study of recitation: how often the model outputs a fragment that repeats existing code verbatim. It found about 0.1 percent of cases, although the distribution is not uniform and recitation happens more often when the file is empty and the model has nothing to work from. In that study, detection of such repetitions had not yet been integrated into the product. Today the switch is there, and that is a real difference from February.
It does not change my process, however. The filter works on long blocks, while I deal mainly with short suggestions that no character threshold will catch. I read the diff just as I did before because that is the only mechanism that covers the entire problem.
Still not Completions and not chat
A year ago, in June 2021, I described an experiment with generating test data through the Completions API. I see that today’s launch is once again putting both things into the same “language models” bucket, so I will separate them once more because GA has not merged anything here.
That was an HTTP call. An endpoint with davinci in the address, a hand-built few-shot prompt, temperature, max_tokens, stop, a text response, schema validation, and only then a file in the repository. Full control over the input, full responsibility for joining the result to the project, and data as the result.
Copilot is on the other side. The prompt is my open file and the comment I write in it. There are no parameters, no key in an environment variable, and no response I can validate because the response is immediately in the editor. The result is not an artifact, but the way my code was created, and once accepted it cannot be distinguished from what I typed with my own fingers.
Today’s announcement does not add a conversation window to Copilot. There is still nowhere to ask the tool why it suggested that particular thing. The file is the entire input channel, and reading the code is the entire verification process.
The risk that remains
The list of things to remember is exactly the same today as it was yesterday. Paying for the product removes nothing from it.
Telemetry and sending context. To receive a suggestion, a fragment of my file leaves for the service. The service also records what I accept and what I reject. There is no offline mode. This is still the main reason why the decision of “where do I enable this?” is more serious than “do I buy this?”
Suggestion security. In August 2021, NYU researchers ran generated code through a set of scenarios built around common vulnerability classes, and a substantial number of programs had a vulnerability: no input validation, mishandling external data, or weak use of cryptography. This is not an accusation, only a description of the mechanism. The model suggests code similar to code it has seen, and code it has seen is sometimes vulnerable. This work should be kept separate from today’s GitHub metric. Both talk about forty percent, but they measure completely different things.
Outdated APIs. Suggestions with patterns that are no longer used in .NET 6, or with synchronous calls on a path that is asynchronous in my code. This compiles, so it can only be caught by eye.
Data that looks real. Generating sample records can produce addresses and numbers that look as if they belong to someone. I stick to documentation domains in fixtures.
The layer below the code. Copilot does not know which image I have in the registry, how much memory the CI agent has, or whether a port is free. When I write an integration test where I start the database with Testcontainers, ghost text very smoothly suggests container configuration using an API incompatible with the pinned package version. It is similar with isolation when running tests in parallel in CI: writing a test faster does not solve a single problem with shared state. The infrastructure remains one hundred percent my responsibility.
Responsibility. GitHub has said this from the start and repeats it today: code written with Copilot’s help belongs to me, and I am responsible for it. A paid invoice does not transfer that responsibility to the provider.
Summary
Today there is one decision to make: I pay $10 per month or stop after the trial. I do not have to do anything until 22 August because my preview access continues to work. I am staying, not because the tool does anything new, but because for several months it has saved me a real, tedious part of the typing and I have developed the habit of rejecting what needs to be rejected.
The habit matters more here than the price. Three rules from the preview remain unchanged: establish the convention before asking for code, read the assertion and locator character by character, run the test locally and check whether it can fail. Leaving technical preview does not change the nature of the tool. It is still predicting the next line from what is visible in the file.
For teams, today’s answer is: not yet. The business plan consists of GitHub’s words “later this year”, so today the only option is to buy individual subscriptions and live with the fact that they are attached to personal accounts. When an offering for organizations appears, there will be a separate question about permissions, seat management, and which repositories it may be enabled in at all.
And one thing I repeat whenever a faster tool appears in my toolbox. The pace of writing code has increased, and that is not the same as an improvement in quality. The point where it is decided whether a test suite means anything still lies in the process, not in the editor. A year ago I closed my 2021 retrospective by saying that language models belonged in my “experiment at the desk” category. After today, I am moving them into the “tool I pay for” category. That is a move by one shelf, not a change of profession.