Pairing in stable VS Code, not the May preview
It is 15 April. In May 2023 Copilot Chat lived in VS Code Insiders, behind a waitlist and the Nightly extension. Today I open the stable editor, the same one where I have had ghost text since June 2022, and on the side I have Chat, generally available since 29 December. Next to chat I have an @playwright/test spec on pin 1.43.1 from 12 April. That is the E2E pairing this post is about. It is not a second intro to Chat, and it is not the issue-to-pull-request bridge Universe promised for 2024.
My thesis for April 2024 is narrow. Pair programming on a browser test looks like this: Chat drafts a step and an assertion, I run npx playwright test, and I reject an invented data-testid. I remain the owner of the assertion, as I was with ghost text in April 2023. Chat is not the owner of the suite. In January I wrote that a loop with a goal is not the owner either. April’s difference is the tool: I do not give the model the goal “fix the test”, I talk about the open spec and the helper beside it.
Here is the state of the workshop today. The runner is Playwright 1.43.1. Component testing is still experimental and is not the subject of this pairing; I wrote about CT in August. I have Chat in stable VS Code. When the helper sits in a different file from the spec, I add @workspace so chat can collect references, and I still decide which file to touch. I do not have a preview of a product that would take a change from an issue to a PR, and I will not pretend that I do.
What will not be here. There will be no sentence that Chat is still on a waitlist. There will be no tutorial of an agent that attaches a patch to an issue on its own. There will be no project init from scratch, because the first Playwright project and the decision Playwright over Cypress have been behind me since 2022. There will be a synthetic login spec, two sidebar proposals, and strikeouts.
What I put into the pair
I open three things and nothing from .env.
The first is login.spec.ts. The happy path is already there: a correct password, getByRole('button', { name: 'Log in' }), an assertion on the dashboard heading. The second is a slice of the POM, LoginPage.ts, because locators should not live in the spec if I use them in two scenarios. The third is a criterion in one sentence: a wrong password leaves the form and shows an alert “Invalid password”.
When Chat only sees the spec, it guesses the POM. So for a question about the second scenario I add @workspace, or I paste the page-class signature if I want a narrower context. I do not paste connection strings, tokens, or real passwords of test users. A pair that needs a secret in chat is not a pair I use.
The question I started this week with was this:
login.spec.ts only has the happy path. Add a wrong-password scenario.
The alert has role alert and the text "Invalid password".
Do not use waitForTimeout. Keep the POM if the method is already there.The answer arrives as a draft in the sidebar, not as a commit.
The pairing loop
The loop has four steps, and none of them is “apply and forget”.
First: Chat proposes a locator and an expect. I read that as a colleague’s review, not as code that is already mine.
Second: I paste the accepted fragment into the spec and run npx playwright test login.spec.ts. Playwright’s auto-waiting stays. I do not ask Chat for waitForTimeout, because a sleep that “fixes” a race hides what we were actually waiting for.
Third: on failure I open the trace. I look for whether the alert was in the DOM, whether it was hidden, whether the text sat in aria-live instead of role=alert. Chat cannot see that. It sees the file.
Fourth: I go back to the sidebar with the error message and one sentence about what not to do. I do not write “fix it somehow”. I write ”login-error testid does not exist, use the role”. The tighter the return, the less room Chat has for a second invention.
This is not an agent with a tool loop. This is a pair in which the runner and the trace are mine.
Where Chat lies on E2E
Two proposals from this week, both from the synthetic login, both to strike out.
Strikeout one: an invented data-testid. Chat added page.getByTestId('login-error'). That attribute does not exist in the component, in the POM, or in the DOM from the trace. The model inserted it because in other repositories error alerts have a stable testid, and that is a statistically good guess. The effect is that the test waits for an element that never appears, and fails with a timeout that looks like environment flake. The assertion that passes is getByRole('alert') plus toHaveText('Invalid password'). The role is in the HTML. The testid is in the model’s imagination.
Strikeout two: an API that is not on 1.43. In the second draft Chat proposed toHaveAccessibleName as a shortcut for the same alert. On pin 1.43.1 I do not have that assertion. The runner, not chat, is the source of truth about the API. If I had pasted that line, I would have gotten TypeScript or a runtime error before I got a verdict on the password. I stay with getByRole and toHaveText, which I know from the runner I actually run.
Beside those two there is still the reflex I know from ghost text: an assertion on marketing copy from the login hero instead of on the alert role. An expect(page.getByText('Welcome back')).toBeVisible() passes regardless of the password. That is not an error test. That is a test that the page rendered.
What I do not delegate
Chat speeds up the first draft of a spec. It does not speed up the decision of whether the scenario deserves E2E at all.
A wrong password on the login form is a gate for me, not an overnight run of the whole app, and I drew that line in September 2022. Whether this case should live as a browser test or as a unit on the message reducer is a question for the pyramid, not for the sidebar. Experimental CT does not enter this pairing: I do not mix mounting a component with E2E login only because Chat knows both APIs.
I note SWE-agent from 2 April in one sentence as issue-to-patch research; I do not install it and I do not make it daily work on a spec. Claude 3 stays in browser chat. The E2E pair is Copilot in the editor, because it sees the file I have open.
Summary
In April 2024, pairing on E2E is Copilot Chat GA beside an open Playwright 1.43 spec. The draft is born in the sidebar. Green is checked by the runner. A human signs the merge.
Three sentences I want to be able to read a year from now. Chat in stable VS Code drafts a second login scenario faster than I type the boilerplate myself, and that is a real gain versus May 2023. Two drafts from this week had to be struck out: a testid that does not exist, and an assertion from an API that does not exist on 1.43.1. The pairing loop ends with running the test and the trace, not with an apply button.
What does not follow. It does not follow that I go back to Insiders, because GA on 29 December is exactly what May was missing: the same editor as the rest of the work. It does not follow that Chat can declare a spec green on its own. It also does not follow that the issue-plan-PR bridge already works for me, because I do not have it.
The condition by which I will know the pair has matured is this: two drafts in a row on the same criterion use a role that is in the DOM and an API that is in 1.43.1. Until then I draft with Chat and I merge myself.