k6 and Playwright - load apart from E2E (September 2024)

Green E2E, an unanswered capacity question

It is September 15. My functional Playwright suite is green, login passes on every run, and I still cannot answer one question: what will happen when fifty people log in at the same time. A green E2E test says that the path works for one user in one browser context. It does not say how many such paths the application can handle at once or how long each one will have to wait, and it should not.

In September 2022 I divided tests into three buckets: pull request, main, and night. That division remains unchanged. Today I am adding Grafana k6 to it, and this post is about where k6 stands alongside Playwright. I am not rewriting the pyramid, changing the pull request gate, or replacing the functional suite.

My thesis for September 2024 is narrow. k6, both at the protocol layer and in the k6/browser module that has been stable since June, measures time and load. Playwright 1.47.1 remains the owner of functional assertions. The k6 browser API is roughly similar to Playwright, but it is not a drop-in: I cannot move a script from one tool to the other without rewriting it, and k6 does not replace the E2E suite on a pull request.

Here is the state of the workshop, so that a year from now it is clear what I mean. k6 OSS is 0.53.0 from August 13, the first release with native ES modules. Its JavaScript engine is Sobek, the goja fork that k6 switched to in June. The k6/browser module left the experimental phase in 0.52.0 on June 25, immediately with a fully asynchronous API. The old k6/experimental/browser import still works, but it is supported only until September 23 and is not a pin for anything in this post. Browser tests in Grafana Cloud k6 have been generally available since July 4, while the demo in this post is entirely OSS. Playwright is 1.47.1 from Friday, September 13, a patch for 1.47.0 from September 5, with Chromium 129.0.6668.29 pinned to that version.

What will not be here. There will be no sentence saying that k6 replaces @playwright/test. There will be no new YAML from scratch, because I described the triggers two years ago and today I am adding two jobs to them. There will be no production request count per second, client name, or Grafana Cloud pricing. I am not teaching Gatling or JMeter here. There will be a synthetic application with login, a protocol script for its API with a p95 threshold, one k6/browser script that clicks through the form, and a Playwright test for the same path.

All numbers in this post come from one four-core machine running the application, k6, and Chromium at the same time. This illustrates the mechanism. It is not a benchmark. I will return to that caveat in the traps, because it is itself one of them.

Two k6 layers

The synthetic application is deliberately small. The /login/ page has a form with “Email” and “Password” fields and a “Sign in” button. The form calls POST /api/login, which checks the password and sets a session cookie, and after success the browser moves to /account/, where it finds a “Welcome” heading. The password is checked with PBKDF2, and the whole experiment sits here because the application has two modes. The first calculates the hash in a thread pool (crypto.pbkdf2). The second calculates it synchronously (crypto.pbkdf2Sync), on the same thread that handles all requests. The second mode is a regression inserted deliberately, the kind that arrives in real code with a small refactor. Functionally, both modes are identical: the same password, the same status, and the same cookie.

The first layer is protocol. k6 sends HTTP requests without a browser, so a virtual user, or VU, is cheap: it is a loop in JavaScript, not a separate process. Here is the complete perf/login-api.js:

import http from 'k6/http'
import { check, sleep } from 'k6'

const BASE_URL = __ENV.BASE_URL || 'http://localhost:3000'

const profiles = {
  smoke: { executor: 'constant-vus', vus: 2, duration: '30s' },
  load: {
    executor: 'ramping-vus',
    stages: [
      { duration: '30s', target: 50 },
      { duration: '1m', target: 50 },
      { duration: '15s', target: 0 },
    ],
  },
}

export const options = {
  scenarios: { login: profiles[__ENV.PROFILE || 'smoke'] },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<300'],
  },
}

const body = JSON.stringify({ email: 'qa@example.test', password: 'Haslo123!' })
const params = { headers: { 'Content-Type': 'application/json' } }

export default function () {
  const res = http.post(`${BASE_URL}/api/login`, body, params)
  check(res, { 'status 200': r => r.status === 200 })
  sleep(1)
}

Several details here are deliberate. I select the traffic profile with the PROFILE variable: smoke is two users for 30 seconds, while load is a ramp to fifty, one minute on the plateau, and a ramp down to zero. sleep(1) is think time between logins. Without it, fifty VUs become fifty breathless loops. The thresholds in options.thresholds, not check, decide the result of the run. When a threshold is crossed, k6 ends the run with exit code 99, which means a red step in CI. The status 200 check is only a counter in the report in this script, and in the traps I will show why that distinction matters.

Smoke on the synchronous variant passes without strain: p95 is 42.55 ms, there are 58 requests, all with status 200, and the exit code is 0. Two users do not see the regression, just as the E2E test does not. Load on the same variant looks different:

     ✓ status 200

     checks.........................: 100.00% ✓ 2471      ✗ 0
   ✗ http_req_duration..............: avg=683.18ms min=35.72ms med=871.42ms max=2.83s    p(90)=905.8ms  p(95)=915.45ms
   ✓ http_req_failed................: 0.00%   ✓ 0         ✗ 2471
     http_reqs......................: 2471    23.473169/s
     vus_max........................: 50      min=50      max=50

The run ends with thresholds on metrics 'http_req_duration' have been crossed and exit code 99. Every one of the 2,471 requests received status 200, none failed, and p95 reached 915.45 ms against a 300 ms threshold. On the thread-pool variant, the same profile produces a p95 of 40.08 ms and exit code 0. In the synchronous variant, each login occupies the only request-handling thread for several dozen milliseconds. With two users this is invisible, but with fifty the requests form a queue.

The second number in that report matters as much as p95: 2,471 requests in the same period in which the fast variant handled 4,003. A VU ramp in k6 is a closed model. Each virtual user waits for a response before sending the next request, so a slower application receives less traffic and reduces its own load. That is an honest answer to the question about fifty concurrent sessions. The question of whether an application can handle a specified number of requests per second needs an open model, which in k6 means the constant-arrival-rate executor. It sends requests at a constant rate regardless of how quickly the system responds. This post is not about RPS, so I am staying with VUs.

The second layer is the browser. k6/browser drives real Chromium and collects frontend metrics: Web Vitals such as browser_web_vital_lcp, browser_web_vital_fcp, and browser_web_vital_ttfb, as well as the timings and errors of requests sent by the browser itself. A VU is no longer cheap here because each one gets its own Chromium instance. Here is the only k6/browser script in this post, perf/login-browser.js:

import { browser } from 'k6/browser'
import { check } from 'k6'
import { Trend } from 'k6/metrics'

const BASE_URL = __ENV.BASE_URL || 'http://localhost:3000'
const loginDuration = new Trend('login_ui_duration', true)

export const options = {
  scenarios: {
    ui: {
      executor: 'constant-vus',
      vus: 2,
      duration: '1m',
      options: { browser: { type: 'chromium' } },
    },
  },
  thresholds: {
    checks: ['rate==1.0'],
    browser_web_vital_lcp: ['p(95)<1000'],
    login_ui_duration: ['p(95)<1500'],
  },
}

export default async function () {
  const page = await browser.newPage()
  let landed = false
  try {
    await page.goto(`${BASE_URL}/login/`)
    await page.locator('#email').fill('qa@example.test')
    await page.locator('#password').fill('Haslo123!')

    const start = Date.now()
    await Promise.all([
      page.waitForNavigation(),
      page.locator('button[type="submit"]').click(),
    ])
    loginDuration.add(Date.now() - start)
    landed = page.url().endsWith('/account/')
  } finally {
    check(landed, { 'iteration reached /account/': ok => ok })
    await page.close()
  }
}

Web Vitals measure page loads, but I am also interested in the time from clicking “Sign in” to landing on /account/, including the API call. That is why I add a custom Trend metric named login_ui_duration, with a threshold on p95. I capture the navigation after the click with Promise.all and page.waitForNavigation() because k6 has no assertion that waits for the URL by itself, like expect(page).toHaveURL in Playwright. The check sits in finally and has a rate==1.0 threshold. I will explain why it sits there in the traps, because this was the most expensive lesson of the week.

k6 0.53.0 does not bring its own browser. It looks for Chrome or Chromium on the system, and the path can be overridden with the K6_BROWSER_EXECUTABLE_PATH variable. Locally, I point it to Chromium 129.0.6668.29, which Playwright 1.47.1 installs in its browser directory, so that both tools run on the same engine build. Two users for one minute on the fast variant produce 174 iterations, all ending on /account/, an LCP p95 of 38.49 ms, a login_ui_duration p95 of 257 ms, and exit code 0. On pages this simple, LCP and FCP are identical because the largest element appears with the first render of the content.

If I needed more browsers than one machine could handle, Grafana Cloud k6 offers browser tests with multiple VUs, a browser timeline, and screenshots. Since 0.53.0, the same script can be sent there with k6 cloud run. I am not giving pricing, and OSS is entirely sufficient for this post.

Why I do not merge it with Playwright

On the same synchronous variant where k6 crossed its threshold, Playwright 1.47.1 passes without complaint. Here are both tests from tests/login.spec.ts:

import { test, expect } from '@playwright/test'

test('login moves to the account page', async ({ page }) => {
  await page.goto('/login/')
  await page.getByLabel('Email').fill('qa@example.test')
  await page.getByLabel('Password').fill('Haslo123!')
  await page.getByRole('button', { name: 'Sign in' }).click()

  await expect(page).toHaveURL(/\/account\/$/)
  await expect(page.getByRole('heading', { name: 'Welcome' })).toBeVisible()
})

test('a wrong password stays on login with a message', async ({ page }) => {
  await page.goto('/login/')
  await page.getByLabel('Email').fill('qa@example.test')
  await page.getByLabel('Password').fill('wrong-password')
  await page.getByRole('button', { name: 'Sign in' }).click()

  await expect(page.getByRole('alert')).toHaveText('Invalid email or password')
  await expect(page).toHaveURL(/\/login\/$/)
})

Running 2 tests using 1 worker

  ✓  1 [chromium] › login.spec.ts:3:5 › login moves to the account page (241ms)
  ✓  2 [chromium] › login.spec.ts:13:5 › a wrong password stays on login with a message (211ms)

  2 passed (866ms)

Both tools tell the truth and answer different questions. Playwright checks whether the path works: whether a good password takes me to the account page, and whether a bad one shows a message and leaves me on login. k6 checks what that path costs under traffic. If I replaced either result with the other, I would lose information in both directions.

Playwright has been my functional suite since the decision from April 2022 and the empty project from May of that year. The reasons have not changed. Assertions wait for page state by themselves, locators follow roles and labels and my page objects rest on them, three engines share one API, and a trace from CI shows the step, DOM, network, and console for a red test. Trace Viewer is a living tool here, not an accessory: 1.47.0 added filters to the network tab, and Friday’s 1.47.1 fixed response body display in that tab. One scenario from this suite runs on the pull request as smoke, while the full matrix runs at night, exactly as I decided in 2022.

k6 has different strengths and different boundaries. A script does not run in Node but on the Sobek engine, so k6 does not resolve packages from node_modules like Node does, and importing @playwright/test ends before anything starts:

level=error msg="GoError: The moduleSpecifier \"@playwright/test\" couldn't be recognised as something k6 supports.\n" hint="script exception"

That is exit code 107, meaning a script error. I then pasted the steps from the first test, without the assertions, into a k6 skeleton with browser.newPage() at the beginning and page.close() in finally, changing only the address in page.goto to a full URL. k6 opened the browser, loaded the page, and failed on the first locator with TypeError: Object has no member 'getByLabel'. A separate variant with page.locator and a Playwright-style assertion ended with ReferenceError: expect is not defined. Both runs ended with exit code 0 because the script had no thresholds, and the summary still showed an LCP of 24.89 ms because page.goto had time to execute.

The module documentation for version 0.53 states this honestly: the browser API aims for rough compatibility with Playwright for Node.js, and because k6 does not run in Node.js, it will differ somewhat. In practice, those differences mean no getByRole, getByLabel, or other getBy* methods on the page and locator, no built-in expect, and one engine, because type: 'chromium' is required in the scenario options and is the only available value. The k6chaijs library from jslib provides Chai-style assertions, but its expect does not wait for page state like Playwright’s expect. In the other direction, k6 has things Playwright does not provide directly, such as page.throttleCPU() and page.throttleNetwork().

The k6 documentation page about the Page Object Model is especially tempting because it says existing page objects are easy to reuse. The example on that page uses page.locator with a data-testid attribute. My Playwright page objects use getByRole and getByLabel, so easy reuse for me means rewriting every locator. That is why the k6 script in this post has CSS selectors based on HTML IDs, while the Playwright test has roles and labels. The syntax looks familiar: page.goto, locator, fill, click. I do not copy the script 1:1 or pretend that I maintain one piece of code in two tools.

The word “parallel” also means something different here than it did in October 2021. There, parallelism means framework threads, VSTest processes, and pipeline jobs, while strategy: parallel in Azure or max-parallel in a GitHub Actions matrix says how many machines I divide the suite across so it finishes sooner. VUs in k6 are simulated users in one process on one machine. Their number is part of the question, not a way to get the answer faster. In 2021, I limited max-parallel when tests hit a shared environment that would not survive twenty clients. k6 turns that sentence into a measurement: how many clients the environment can handle and how long each of them waits.

A year ago I wrote that AI is a production line, not a new level of the pyramid. k6 is the same. It does not flatten the pyramid or add a floor to it, because the pyramid is a picture of execution cost and feedback speed, while load is a separate question asked beside it, on a different trigger.

Where in the triggers

The triggers come from September 2022, and I am not changing a word in them. I am only answering which bucket receives each k6 layer.

The pull request gets nothing. pr-gate remains the only required check and has the same composition as two years ago: unit tests, consumer contracts, and one application smoke test. I admit only deterministic work to this bucket, and timing on a shared runner is not deterministic. A gate that sometimes lies stops being a gate and becomes a formality, and that is not why I made it leaner two years ago.

On main after the merge, I run protocol smoke: the same perf/login-api.js with the smoke profile, two users for 30 seconds, and the same thresholds. In this job I start the application in the same way as in the e2e job from August 2022, with a Compose file and a loop over /health. This smoke test would not catch the synchronous-hash regression, and it is not supposed to, because that regression is invisible with two users. It catches two other things. The first is a script that has gone stale: a changed endpoint path or a different request shape to which the API responds with an error, either of which the http_req_failed threshold will stop. The second is a regression already visible to one user, such as a query that suddenly takes half a second. Without this job, I would learn about a broken script from the nightly load run, and a nightly load run that is red because of a typo in the script teaches the team that a red nightly load run means nothing.

Load and the browser run at night, in a separate job beside the nightly E2E matrix rather than inside it, and that job is not required. The nightly script combines both layers in the way the k6 documentation describes the hybrid approach: many protocol VUs generate load while a few browser VUs measure what a user sees at the same time. The documentation recommends starting with a small browser share, around ten percent of VUs or less. Mine is two out of fifty-two. Thanks to the native modules in 0.53.0, perf/nightly.js only re-exports functions from the two scripts and assembles scenarios from them:

export { default as api } from './login-api.js'
export { default as ui } from './login-browser.js'

export const options = {
  scenarios: {
    api: {
      executor: 'ramping-vus',
      exec: 'api',
      stages: [
        { duration: '30s', target: 50 },
        { duration: '1m', target: 50 },
        { duration: '15s', target: 0 },
      ],
    },
    ui: {
      executor: 'constant-vus',
      exec: 'ui',
      vus: 2,
      startTime: '30s',
      duration: '1m',
      options: { browser: { type: 'chromium' } },
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<300'],
    'checks{scenario:ui}': ['rate==1.0'],
    browser_web_vital_lcp: ['p(95)<1000'],
    login_ui_duration: ['p(95)<1500'],
  },
}

k6 reads options only from the file I run, so the thresholds are repeated here. I narrowed the threshold on checks to the ui scenario, which means the guard in finally. The http_req_failed threshold watches API response status with a budget below one percent, while a rate==1.0 threshold on all checks would cancel that budget after the first 500 response among several thousand. The browser starts after 30 seconds, when the ramp reaches fifty VUs, and measures the full minute on the plateau.

All thresholds pass on the fast variant, but the numbers still move: API p95 rises from 40.08 ms without the browser to 74.06 ms, LCP rises from 38.49 to 63.98 ms, and login_ui_duration rises from 257 to 298.95 ms. Part of the difference is the load, and part is the two Chromium instances taking CPU away from the application on the same machine. On the synchronous variant, the same file produces:

     ✓ status 200
     ✓ iteration reached /account/

   ✗ browser_web_vital_lcp..........: avg=1.68s    min=643.1ms  med=1.57s    max=3.71s    p(90)=2.41s    p(95)=2.47s
     browser_web_vital_ttfb.........: avg=1.66s    min=630.8ms  med=1.55s    max=3.69s    p(90)=2.39s    p(95)=2.45s
     checks.........................: 100.00% ✓ 2489      ✗ 0
     ✓ { scenario:ui }..............: 100.00% ✓ 20        ✗ 0
   ✗ http_req_duration..............: avg=684.87ms min=36.15ms  med=874.23ms max=2.32s    p(90)=941.25ms p(95)=953.73ms
   ✓ http_req_failed................: 0.00%   ✓ 0         ✗ 2469
     http_reqs......................: 2469    23.438002/s
   ✗ login_ui_duration..............: avg=3.45s    min=2.93s    med=3.49s    max=3.64s    p(90)=3.58s    p(95)=3.6s

The run ends with thresholds on metrics 'browser_web_vital_lcp, http_req_duration, login_ui_duration' have been crossed and code 99. This is the point of the entire post for me. The browser guard is at 100 percent because every one of the 20 iterations reached /account/, and none of the 2,469 API requests failed. Functionally, everything works, just as it does in Playwright. At the same time, the /login/ and /account/ pages, both static, have an LCP p95 of 2.47 s, while login from click to account takes 3.6 s at p95. The static pages wait because the synchronous hash blocks the only request-handling thread and everything else queues behind it. A TTFB p95 of 2.45 s says this directly: the browser is waiting for the first byte, not for rendering.

In GitHub Actions, I add both jobs to the sketch from 2022, under the existing jobs:. As before, I omit checkout and result publishing so that only the triggers are visible. I keep the k6 installation because it carries the pin.

jobs:
  # pr-gate, integration, and nightly unchanged

  perf-smoke:
    if: github.event_name == 'push'
    runs-on: ubuntu-22.04
    steps:
      - run: docker compose -f docker-compose.e2e.yml up -d
      # /health loop as in the e2e job from August 2022
      - uses: grafana/setup-k6-action@v1.1.0
        with:
          k6-version: '0.53.0'
      - run: k6 run -e PROFILE=smoke perf/login-api.js
        env:
          BASE_URL: http://localhost:5000

  load:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-22.04
    steps:
      - uses: grafana/setup-k6-action@v1.1.0
        with:
          k6-version: '0.53.0'
          browser: true
      - run: k6 run perf/nightly.js
        env:
          BASE_URL: ${{ vars.PERF_BASE_URL }}

The runner is pinned to ubuntu-22.04, not ubuntu-latest, for the same reason as two years ago. k6-version is also pinned because without it the action installs the latest k6 and the nightly job would move by itself to the next release, the one where the old browser module disappears. The browser is worse. On Linux, browser: true in grafana/setup-k6-action 1.1.0 checks google-chrome --version and installs google-chrome-stable from Google’s repository only if Chrome is missing. Chrome is present in the Ubuntu hosted runner image, so k6 is pinned while the browser comes from the runner image and will change without my commit. If I also want to pin it, there is a grafana/k6:0.53.0-with-browser image in which Chromium comes with k6.

The nightly load targets a stable environment through vars.PERF_BASE_URL, not an application started on the same runner. Smoke on main can share a machine with the application because two protocol users cost almost nothing. Fifty VUs and two Chromium instances already cost a lot: in my measurement, adding two browser VUs to the same profile raised API p95 from 40.08 to 74.06 ms. A load generator and an application on one machine measure each other.

In 2022, I wrote that a nightly bucket without a name on the calendar dies in three weeks. A nightly load run dies faster because its red does not look like an error, but like a number that is easy to dismiss as noise. That is why this job has an owner who reads the thresholds in the morning before anyone says, “probably the runner.”

September traps

Three strikeouts from this week. In each one, k6 showed me a result that looked better than it should have.

Strike one: twenty browsers on one runner. To learn how many browser VUs the machine could handle, I ran only the k6/browser script against the fast application variant, for 30 seconds each with 1, 5, 10, and 20 VUs. Every two seconds, I sampled whole-machine CPU utilization and the number of Chromium processes.

Each VU means roughly ten Chromium processes. From ten VUs onward, the iteration count stays in place: doubling the VUs from ten to twenty produced 180 iterations instead of 176, but doubled LCP and login p95, and a repeat showed the same thing. During this time, the application handles about six logins per second, using less than a quarter of one core. Chromium on the generator is the bottleneck, not the application. Memory was not the problem: with twenty VUs, available memory dropped by about 2.2 GB, or roughly 110 MB per browser. CPU ran out much earlier.

The worst part of this result is that it looks credible. All thresholds passed, an LCP of 366.82 ms is a reasonable number, and nothing in the report says that I am measuring the k6 machine. With stricter thresholds this run would be red, and the browser on the generator, not the application, would be at fault. That is why protocol stays cheap and broad, while the browser stays narrow: a few VUs that measure user experience rather than generate traffic.

Strike two: a functional assertion in the load script. I distinguish two kinds of checks. A functional assertion asks whether the application does what it should: whether a bad password produces a message, whether the heading says “Welcome”, or whether the response has the right shape. A measurement guard asks only whether the iteration followed the path whose time I am measuring. The first belongs in Playwright and contract tests. The second must be in k6, with a threshold.

A functional assertion in k6 loses in both variants. I added a check to the API script saying that the response contains a token. The application does not return a token in the body but sets a cookie, so the check failed twenty times out of twenty. checks showed 50 percent, the thresholds passed, and the exit code was 0. Without a threshold on checks, an assertion in k6 is a counter in a report that someone may or may not read. With a threshold, the same run would be red, but for a reason unrelated to load: in the morning I would see a red nightly load run and look for a bottleneck where the API contract had changed. Two questions in one run produce a red result I cannot attribute.

The measurement guard is necessary for a reason I found by accident. The first version of the browser script had the check at the end of the try block, after navigation. To see what would happen when the script crashed, I changed one locator to page.getByLabel('Email'), which k6 does not have. Each of the 18 iterations ended with an exception in the log, while the summary looked like this:

   ✓ browser_web_vital_lcp.......: avg=24.31ms  min=19.5ms   med=23.39ms  max=34.7ms   p(90)=28.66ms p(95)=31.29ms
   ✓ checks......................: 0.00% ✓ 0        ✗ 0
     iterations..................: 18    3.539449/s
   ✓ login_ui_duration...........: avg=0s       min=0s       med=0s       max=0s       p(90)=0s      p(95)=0s

The exit code was 0 and all three thresholds were green because in 0.53.0 a threshold on a metric with no samples passes: checks received no result, while a Trend with no samples has a p95 of zero. LCP is real because page.goto had time to execute before the script failed. An exception in an iteration is only logged by k6, which then continues, with no effect on the exit code. After I moved the check into finally, every iteration left a result, including one that crashed, and the same broken script ended with checks at 0 out of 18, a message that the checks threshold had been crossed, and code 99. That is why the guard in the script in this post sits in finally.

In 2019 I wrote that a suite of ten tests everyone can rely on is better than two hundred or three hundred tests nobody trusts. A nightly load run has the same problem in a sharper form. One green run with zero samples is enough for nobody to trust green again.

Strike three: the old k6/experimental/browser import. A spring-style script with the k6/experimental/browser import, synchronous browser.newPage(), and no await before locators still passes on 0.53.0. I get only a warning that the module will disappear after September 23, with a link to the migration guide. The temptation is obvious: change one import line and be done. I did exactly that and got TypeError: Object has no member 'close'. In the stable module, browser.newPage() returns a promise, so without await, the page variable is not a page. The first error came from page.goto, but the exception from page.close() in the finally block masked it, so the log points to the wrong line. The exit code was again 0 because the script had no thresholds.

Migration is not an import change. It means adding await before newPage, goto, fill, click, and close, which is nearly every call. The migration guide states this directly and adds two things that are easy to miss: the core check does not wait for promises, and group does not work with the asynchronous API. I verified the first. An asynchronous predicate passed to the core check ends an iteration on 0.53.0 with check() does not support async functions as arguments and leaves no sample in checks, which leads straight to the trap from the previous strike. That is why my script calculates landed first and passes a regular function to check. An asynchronous check has been available in the k6-utils jslib since August.

According to the announcement, the next k6 release will remove the old module. On 0.53.0, a spring script still runs, and only the k6-version pin in CI separates it from a nightly upgrade. The canonical form is k6/browser with await before nearly every call.

OpenAI o1-preview from September 12 does not write my load scenario: I derive the traffic profile and thresholds from measurements, not from chat.

Summary

In September 2024, k6 measures capacity while Playwright says whether the path works. Both run in their own jobs, and neither pretends to be the other.

Three sentences I want to be able to read a year from now. On synthetic login, the regression from a synchronous password hash was invisible to Playwright and to smoke with two VUs, but visible to a ramp to fifty VUs: p95 was 915.45 ms against a 300 ms threshold, with exit code 99 and every response returning status 200. k6/browser in 0.53.0 is roughly similar to Playwright, but it has no getBy* or expect and knows only Chromium, so I write the script separately and keep a guard in finally, because a threshold on a metric with no samples passes. k6 does not enter the pull request: protocol smoke runs on main, while load with two browser VUs runs at night, in a separate job outside the required checks.

What does not follow. It does not follow that I am deleting Playwright tests or moving them to k6. It does not follow that p95 from one four-core machine says anything about production: it explains the mechanism. It does not follow that I need Grafana Cloud k6, because the whole demo is OSS. And it does not follow that smoke on main protects against a capacity regression, because I have just shown that it does not.

I state the condition under which nightly load will stop being an experiment for me and become a full member of the nightly bucket like this. The generator and the application run on different machines. Two nights in a row on the same application version produce the same verdict for every threshold. A deliberately inserted regression, such as the synchronous hash, makes the night red while Playwright remains green. And someone has that job on the calendar and reads the thresholds in the morning before saying, “it is probably the runner.” Until then, a red nightly load run is a report to investigate, not a verdict.