Whether AI is allowed to speak on a PR
It is 15 June. In April 2023 I reviewed Copilot suggestions in the buffer: gray text, Tab, a notebook of forty-one items. Today the question is different. Whether a model is allowed to leave a comment on a pull request before I click merge, and whether that comment is a job in the pipeline or still a chat I open when I want to.
My thesis for June 2024 is narrow. “AI review” in my QA is a job or a card on the PR: a Copilot Enterprise summary, a bot such as CodeRabbit or PR-Agent in Actions. It is not the ghost text from that April. The model comments on a diff. A human remains the required reviewer and the only merge.
Workshop state. The runner is Playwright 1.44.1 from 23 May. Copilot Chat in the IDE has been GA since 29 December, and in April I wrote about pairing on a spec, not about a PR reviewer. Copilot Workspace has been a technical preview on a waitlist since 29 April and is not a reviewer in CI. Copilot Enterprise has been GA since 27 February: 39 dollars per user, not the same 10 dollars as individual Copilot.
What will not be here. There will be no sentence that the “Copilot code review” product is GA. There will be no Autofix as GA. There will be no Claude 3.5. There will be a synthetic workflow and three strikeouts on a comment on a spec.
What is on github.com without a third-party bot
On 27 February Copilot Enterprise became generally available with things I did not have in April 2023: a structured PR summary and chat about the diff on github.com. Both are on-demand. Neither is a required check on main. The individual 10-dollar subscription does not include this, and mixing those two prices into one “Copilot reviews the PR” is false.
I read the summary as a table of contents, not as a review. I ask the diff chat when I cannot hold three spec files at once, in the same spirit as @workspace in March. I still decide whether an assertion checks anything.
A job in Actions: bots that already exist
Pipeline means a token, a workflow, and a comment under the diff, not a VS Code pane.
CodeRabbit from Product Hunt on 10 July 2023 comments on a PR as SaaS. CodiumAI PR-Agent has been OSS since the 24 July 2023 announcement and comes in as a GitHub Action or a /review and /describe command. Both leave a trace on the PR that the whole team can see. Ghost text is seen only by whoever has the buffer open.
A synthetic job, without a token and without an org name, looks like this for me:
# .github/workflows/pr-agent.yml
name: pr-agent
on:
pull_request:
types: [opened, synchronize]
jobs:
review:
runs-on: ubuntu-latest
permissions:
pull-requests: write
contents: read
steps:
- uses: Codium-ai/pr-agent@main
env:
OPENAI_KEY: ${{ secrets.OPENAI_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}This is “in CI” in the only sense I care about: the comment appears without my clicking in the sidebar. It is not required. It does not merge.
Autofix is not a review of tests
Since 20 March the public code scanning autofix beta suggests a CodeQL fix on a PR (JS, TS, Java, Python). That is a neighbour of November’s security basics, not a reviewer of Playwright assertions. Autofix is not GA. GA is a date in the future relative to 15 June and I will not use it. The beta does not judge whether expect(page.getByRole('alert')) checks anything.
How I pin it so the gate does not lie
AI gets the right to a comment or a label. Required remain tests and a human review, as in the CI strategy and in parallel CI.
Three strikeouts from a synthetic comment on a login spec, not from a client PR.
Strikeout one: “no tests”. The bot missed login.spec.ts because the diff was in the POM. It wrote that the change had no coverage. Coverage was in the file next door.
Strikeout two: “add waitForTimeout”. The same reflex I have been striking out in the IDE for a year. Sleep is not a review.
Strikeout three: “require Copilot review before merge”. That check status does not exist. Requiring it on main would be a lie about the product. Required remains a human.
GPT-4o since 13 May is allowed as an API model behind a bot. It is not a June novelty and not a reason for the bot to get a gate.
Summary
Review in CI is a new channel. Merge ownership stays human. Workspace does not review PRs.
Three sentences for a year from now. Enterprise gives a summary and chat about the diff, on-demand, not as a required job. PR-Agent and CodeRabbit leave a comment from Actions; three from my demo had to be struck out. Autofix is a CodeQL beta, not a review of tests and not GA.
What does not follow. It does not follow that I will never turn on a required check on a bot. It does not follow that I go back to Tab only. It does not follow that Copilot code review as a product has already shipped.
The condition: two bot runs on the same spec do not invent missing tests when the file is next door and do not propose sleep. Until then a comment is a signal, and I sign the merge.