Green E2E, an empty capacity question
It is 15 September. I have a green login on Playwright and I still do not know whether checkout survives fifty parallel HTTP sessions. In September 2022 I split the gate, main, and nightly. Today I add k6 to nightly. I do not rewrite the pyramid and I do not drop E2E from the PR.
My thesis for September 2024 is narrow. k6 (protocol plus stable k6/browser) measures time and load. Playwright 1.47.1 from 13 September stays the owner of functional assertions. The k6 API is roughly like Playwright. It is not a drop-in and it does not replace the E2E suite on a PR.
Workshop state. k6 OSS is on 0.53.0 from 13 August. k6/browser left experimental in 0.52.0 on 25 June; the canonical import is k6/browser, not k6/experimental/browser. Grafana Cloud k6 browser tests have been GA since 4 July; the demo in this post is OSS. The functional runner is Playwright 1.47.1. On 12 September, three days ago, OpenAI announced o1-preview. I leave that as a headline. I do not let it write the load scenario.
What will not be here. There will be no sentence that k6 replaces @playwright/test. There will be no pin of 0.54.0 or 1.47.2. There will be no production RPS and no Grafana Cloud quotes. There will be no Gatling tutorial. There will be a synthetic login API with a p95 threshold and one k6/browser script with page.goto.
Two k6 layers
The first layer is protocol. HTTP, cheap VUs, thresholds on http_req_duration. On a synthetic POST /login I set thresholds: { http_req_duration: ['p(95)<500'] } and iterations, not an assertion that “looks at the dashboard”. A load fail is a crossed threshold, not a red expect with a trace.
The second layer is the browser. k6/browser runs real Chromium, few VUs, frontend metrics. A click and page.goto on a synthetic checkout. That is not a hundred sessions on one GitHub Actions runner. That is a handful of virtual users, because Chromium is not cheap.
Cloud k6 browser since 4 July is an option when I do not want Chromium on the agent. OSS is enough to see that the import and the API live. I am not mixing Testcontainers in as a load engine. Containers from August 2022 raise the UI. k6 loads them in a separate process.
Why I do not fold it into Playwright
Playwright: assertions, trace, POM, three engines, the PR job. Since May 2022 and since the Playwright over Cypress decision, that is my functional suite. k6: load, thresholds, a different runtime (Sobek, not Node). The syntax looks familiar. I do not copy a script 1:1.
Grafana itself writes about rough compatibility with Playwright, not about one hundred percent parity. I do not paste getByRole from a POM into k6/browser and pretend I got a trace. The pyramid does not flatten because two tools share a page.goto.
A k6 VU is not strategy.parallel from October 2021. That post is unit and job parallelism. This is virtual users of load. Folding those two words into one YAML ends with a runner that chokes on Chromium and a report that looks like a flake.
Where it sits in the triggers
Protocol smoke may sit on main: few VUs, short duration, a threshold meant to catch an obvious p95 regression, not capacity planning. Load and browser VUs go on nightly or a separate scheduled run. The required check on the PR stays small. I have that boundary from 2022 and from the process. A job that fires fifty VUs on every push teaches the team to ignore red.
I do not add k6 as required on the Playwright login path. Green E2E says the path works. k6 says whether it still breathes under traffic.
September traps
Three strikes from this week.
Strike one: too many browser VUs on one runner. Twenty Chromiums on a hosted agent is not a capacity test of the app. It is an OOM test of the agent. Protocol stays cheap. Browser stays narrow.
Strike two: a functional assertion in the load script. A check that the page contains “Welcome”, next to p95 on the login API, mixes two questions. Either the path works (Playwright) or p95 holds (k6). Both in one script give a red I cannot assign.
Strike three: the old k6/experimental/browser import. Experimental support runs to 23 September. On 15 September that module still exists. The pin is k6/browser from 0.52. If I copied a spring script, I would get a deprecation before I got a metric.
o1-preview since 12 September does not write my scenario. Three days of preview, limits, a different cost. I leave the headline. I write the load scenario, with a threshold I can defend.
Summary
In September 2024 k6 measures capacity. Playwright says whether the path works. Both on their own jobs.
Three sentences for a year. Protocol on synthetic login fails when p95 leaves the threshold, even when E2E is green. k6/browser on 0.53.0 is a handful of Chromium VUs, not a suite replacement. Required on the PR stays small; load sits on nightly.
What does not follow. It does not follow that I delete Playwright. It does not follow that 0.54.0 already sits here. It does not follow that o1 optimises performance.
Maturity condition: two nightlies in a row show the same protocol threshold and do not shout “that is an E2E flake” when k6 is red and Playwright is green.