Pairing in stable VS Code, not the May preview
It is April 15, and there are two things on my screen. On the left is the auth/login.spec.ts spec, and on the right is the Copilot Chat sidebar. I need nothing else to describe what pair programming on browser tests looks like for me this month.
It is worth seeing how much has changed since the last time I wrote about this chat. Last May, half of the post was a set of entry instructions: the Insiders channel, the Nightly extension, joining a waitlist, and waiting. Today I open the same stable editor where since June 2022 I have had gray text in front of the cursor, click the icon in the sidebar, and the chat is simply there. It became generally available to individuals and organizations on December 29, so for the first time I am writing about it without a paragraph on how to get it.
There is a version pin on the other side of the screen too, and it matters more for this post. The runner is @playwright/test version 1.43.1 from April 12, a patch for 1.43.0 from April 4 that closed two UI mode regressions. I give the version down to the third number because the entire second half of this post is about the chat not knowing which version I have installed, while I need to know.
My thesis for April 2024 is narrow, and I will defend it throughout this text. Pairing on E2E means generally available Copilot Chat beside an open Playwright 1.43 spec: the chat drafts a step and an assertion, I run npx playwright test, and I reject a made-up data-testid. That is all. This is not a second introduction to the chat because the introduction was in May. Nor is it the bridge from issue through plan to pull request announced at Universe in November as Copilot Workspace: I cannot use it myself, I do not have a link to it, and I do not intend to describe someone else’s slides. A month ago, I untangled the naming collision between that product and the @workspace agent in the chat window in a note. Today I am writing about something smaller and much more routine: what happens between the sidebar and the runner.
I remain the owner of the assertion. This is not a statement of worldview, only a description of who in this pair has access to the evidence. The chat sees the file I have open. It does not see the DOM of the running application, it does not see the trace, and it does not know whether the error message reached the browser at all. I therefore review its draft exactly as I have reviewed an individual inline suggestion since last April, except that the object of the review is slightly larger: not one line, but a step plus an assertion.
Here is my setup today, so that a year from now it will be clear what I am talking about. Runner 1.43.1. Chat in stable VS Code since December 29, on the same subscription I pay for the gray text. The @workspace agent that searches the repository has been in GitHub’s changelog since November 30, and I call it when the helper is in a different file from the spec. Component testing is still experimental and is not the subject of this pairing; I wrote about it in August. Claude 3 has been a real competitor since March 4, but for a browser tab, not for pairing in the editor. The model behind the chat changes without my involvement anyway: a newer GPT-4 Turbo snapshot appeared on April 9, and nothing follows from that for this post because the only version that settles anything here is the runner’s. The Cursor editor is still not my daily tool.
What this post will not contain. There will be no claim that the chat is behind a waitlist because it has not been since December. There will be no tutorial for an agent that completes a patch from an issue by itself. There will be no project initialization or tool-choice debate because both my first Playwright project and the decision to use Playwright instead of Cypress date back to 2022. There will be a synthetic login from the repository of examples for this blog, one criterion, five drafts from the sidebar, and three rejections.
What I put into the pairing context
The pairing begins with what the chat has in front of it, and this is the only part of the work over which I have full control. I open three things, and none of them comes from .env.
The first is the spec. The happy path is already there:
import { expect, test } from '@playwright/test'
import { LoginPage } from '../pages/LoginPage'
test('logging in with the correct password opens the dashboard', async ({
page,
}) => {
const login = new LoginPage(page)
await login.open()
await login.signIn('user@example.test', 'correct-password')
await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible()
})The second is the pages/LoginPage.ts page object, with locators in the page class, following the convention I established during the POM refactor. It has the email, password, and submit fields and a signIn method. It has nothing that can read an error message, and that is exactly the gap I am closing this week.
The third is the criterion, pasted into the chat as one sentence from my note: a wrong password leaves the user on the form and shows an alert with the text “Invalid password”.
When the chat sees only the spec, it guesses the page object. It guesses surprisingly well because the LoginPage name in the import tells it a lot, and it guesses wrong on the one detail that always costs the most: it adds methods that do not exist in that class. That is why, when I ask for a second scenario, I either call @workspace so that the chat can pull in the class file itself, or paste only the signature when I want narrower context and a shorter response. Both routes have the same boundary, and it is worth stating it plainly: @workspace indexes the repository, not the DOM tree of the running page.
What I do not provide: connection strings, tokens for the test environment, or real passwords for service accounts. The login in this demo is synthetic, and the user is named user@example.test because a pairing workflow that needs a secret pasted into the chat window is not one I use.
The question I started this week with fit into four lines:
auth/login.spec.ts only has the happy path. Add a wrong-password scenario.
The message has the alert role and the text "Invalid password".
Do not use waitForTimeout. If a locator is missing, add it in pages/LoginPage.ts.
Show only the diff. Do not write to any files.The last line is not there by accident. The answer should arrive as text in the sidebar, not as a change in the file, because the first step in this loop is reading, not accepting.
The pairing loop has four steps
None of these steps is called “apply and forget”, and that is the only thing worth remembering from this entire section.
Step one: I read the draft. The chat proposes a locator and an expect, usually within a dozen or so seconds and usually in a shape that looks like my own code because it saw the neighboring test. I treat it like a colleague’s comment on a pull request, not like code that is already mine. The difference lies in the default response: under someone else’s comment, the default response is “I will check”; under an “apply” button, the default response is “yes”.
Step two: I run the test. The accepted fragment lands in the spec, and I immediately run npx playwright test auth/login.spec.ts. Playwright’s auto-waiting remains untouched. I do not ask the chat for waitForTimeout, and I do not accept it when it appears on its own. A sleep that “fixes” a race is not bad because it does not work. It is bad because, when it happens to work, nobody will ever again ask what exactly we were waiting for.
Step three: I read the trace. After a red result, I open the trace. On 1.43, my configuration uses a mode introduced in this exact version:
import { defineConfig } from '@playwright/test'
export default defineConfig({
use: {
trace: 'retain-on-first-failure',
},
})It records a trace for the first run of each test and does not record one for retries, then keeps the file when the test ultimately fails and deletes it when it passes. For this pairing, it is a good default because I am interested in the run where the chat’s draft failed for the first time, not the third attempt after a retry. In the trace, I look for three things: whether the message reached the DOM at all, whether it did but was hidden, or whether it reached a container with aria-live instead of an element with the alert role. The chat sees none of them. It sees the file as it was when I asked.
Step four: I return with the error, not a request. I return to the sidebar with something specific: I paste the runner’s message and add one sentence about what not to do.
Failed: getByTestId('login-error') cannot find the element.
That attribute is not in the application. Use the alert role.I do not write “fix it somehow” because a request without substance returns a draft without substance: the model will replace one made-up locator with another, equally made-up one, and it will do so with the same confidence as the first time. The tighter the feedback, the less room remains for another invention. My pairing loop has a budget of two returns. On the third, I close the window and add the assertion myself because, at that point, reading more drafts costs more than writing two lines.
The loop therefore ends at the runner and the trace, and both belong to me. This is the entire difference between pairing and an agent with its own tool loop: an agent completes the goal on its own, while a pairing partner returns text and waits for the verdict of someone who has access to the evidence.
Where Chat lies in E2E
Five drafts for one criterion, two pasted, three crossed out. I describe the rejections separately because each belongs to a different category of error and, more importantly, each is caught at a different point in the process.
First rejection: a data-testid that does not exist. The first draft looked like this:
await login.signIn('user@example.test', 'wrong-password')
await expect(page.getByTestId('login-error')).toBeVisible()The login-error attribute is not in the form component, it is not in the page object, and it is not in the tree I see in the trace. The model added it because error messages in public code very often have a stable test attribute, and statistically it is a good guess. The runner answers it like this:
1) [chromium] › auth/login.spec.ts:14:3 › logging in with the wrong password shows a message
Error: Timed out 5000ms waiting for expect(locator).toBeVisible()
Locator: getByTestId('login-error')
Expected: visible
Received: <element(s) not found>This is where the true cost of this class of error lies, and it is not in the five seconds lost. This result does not look like an invention. It looks like an environment flake. The test waits for an element that will never appear, fails after a timeout, and lands in the report beside real network timeouts, exactly where the team has learned not to look too closely. The assertion that passes is shorter and contains nothing invented:
await expect(login.error).toHaveText('Invalid password')login.error is getByRole('alert') added to the page object because the form writes the message into a container with that role. The role is in the application’s HTML. The test ID was in the model’s imagination.
Second rejection: an assertion that is not on my pin. The second draft shortened the same check with an accessibility assertion, and in the same response the chat also added a role check to it:
await expect(login.error).toHaveAccessibleName('Invalid password')On 1.43.1, neither of those assertions is available to me, so the compiler responds before the browser even starts:
Property 'toHaveAccessibleName' does not exist on type 'LocatorAssertions'.This is the cheapest of the three rejections and the most instructive because it shows what the model’s “knowledge” of Playwright really is. It is a probability distribution over documentation from many versions, not a reading from my package.json. My project has one version, the one in the lock file, and the runner, not the chat window, is the source of truth about what exists in the API. On this point, TypeScript is sometimes a faster reviewer than I am.
Third rejection: an assertion that always passes. Instead of the error message, the third draft asserted the heading at the top of the login page:
await expect(page.getByText('Welcome back')).toBeVisible()This text is on the page before anything is entered and remains there after a wrong password is entered, so the assertion stays green regardless of what login does. This is not a wrong-password test. It is a test that the page rendered.
These three rejections differ in who catches them, and that difference matters more to me than their content. The runner catches the first at the cost of one timeout. The compiler catches the second at the cost of a second. Nothing but a human review will catch the third because it is green from the first run and will stay green forever, including when the form stops showing any message at all. That is why the review remains mine and why this is the same work I described a year ago for inline suggestions: a green result has never been proof that an assertion makes sense.
What I do not delegate
The chat speeds up the first draft of a spec, and I do not pretend that this is a small thing. The second login scenario appears in the sidebar faster than I can type the import, page object constructor, and three lines of boilerplate myself. It does not, however, speed up any of the decisions that make this test suite worth anything.
The first decision: whether this scenario deserves an E2E test at all. A wrong password on the login form is a pull request gate for me, not part of a nightly run of the entire application, and I set out that boundary in September 2022. Whether the same behavior should also live as a unit test on the message reducer is a question for the pyramid, not the sidebar. The chat will happily write both variants for me and will never say that one of them is enough.
The second: what stays outside the pairing. Component testing is still experimental and does not belong in this spec. I do not mix mounting a component with end-to-end login merely because the chat knows both APIs and will gladly show both; I wrote about where that boundary lies in August, and I have not changed my mind since then.
The third: what I note and what I do not add to my setup. I mention SWE-agent, announced on April 2, in one sentence as research into the path from issue to patch, not as a tool I place beside the spec. Claude 3 stays in a browser tab: to answer the question from the second section, I would have to paste the spec, page object, and note into it and then copy the answer back into the files, returning to the context switching that the chat in the editor has just removed. Copilot is my E2E pairing partner not because it is smarter, but because it sees the file I have open.
Summary
In April 2024, pairing on E2E means generally available Copilot Chat beside an open Playwright 1.43 spec. The draft is created in the sidebar. The runner decides what is green. A human signs off on the merge.
Three sentences I want to be able to read a year from now. The chat in the stable editor drafts the second login scenario faster than I write the boilerplate myself, and that is a real gain over last May, when I first had to get access to the same tool. Out of five drafts for one criterion, I pasted two and rejected three: a test ID that is not in the application, an assertion that is not in 1.43.1, and a green check of a heading that is always on the page. The loop ends when I run the test and read the trace, not when I click the “apply” button.
What this does not mean. It does not mean that I am returning to Insiders because general availability on December 29 gave me exactly what was missing in May: the same editor as the rest of my work. It does not mean that the chat can decide on its own that a spec is green because tools caught only two of the three rejections, while a human caught the third. Nor does it mean that the bridge from issue through plan to pull request already works for me because I do not have it and do not know when I will.
Here is how I will know this pairing has matured: two drafts in a row for the same criterion will use a role that really exists in the DOM and an API that really exists in 1.43.1, while the only surprise in the runner will be the application’s behavior, not the draft’s contents. Until then, I draft with the chat and merge on my own.