ChatGPT (GPT-3.5) for QA - drafting test cases

Six weeks after November 30

OpenAI released ChatGPT on November 30. When I wrote my 2022 review, the release was fifteen days old, and I said plainly that it was a headline, not a tool I had been working with for a year. I also promised to return to the subject in a separate post once I had something beyond a first impression to share. It is January 15, six weeks have passed, and I can now name one place where it has genuinely become part of my work.

That place is not the editor or the pipeline. It is drafting test cases for a user story I have just received, in a browser window next to Jira and the tab with the acceptance criteria.

I want to set the boundary right away because it has been the first thing to blur in conversations over the past few weeks. I am writing about OpenAI’s chat product in the browser, with free access and a model from the GPT-3.5 family underneath. I am not writing about calling it from code. I have no key here, no POST, and no library to install. Nor is this Copilot, which has been sitting in my editor since last year, suggesting the next line in grey text. There, the prompt is my open file and the result is code. Here, the prompt is a user story and the result is a list of sentences in Polish that I read and cross out.

I need to draw this contrast because I have a post on the blog that looks similar from a distance. In June 2021, I generated test data through the Completions API: a waitlist, a key in an environment variable, davinci in the endpoint address, a handwritten few-shot prompt, temperature and max_tokens to configure, followed by schema validation and a file in the repository. There, the result was an artifact that went into a pull request. Here, the result is a draft that goes into my head and my checklist, and the difference in the interface matters less than the difference in what remains afterward.

What I paste and what I do not

I paste three things and nothing more.

The user story, or a summary in my own words. The acceptance criteria exactly as written in the ticket. Context constraints, meaning the user role, the channel, the browsers from the matrix, and the data available to me in the test environment.

I do not paste logs, database dumps, email contents, account identifiers, or anything that came from the production environment. The rule I wrote down in 2021 when working with Completions applies here without a single change, and even more strongly, because a chat window in the browser makes pasting “just this one stack trace” much more tempting than a Python script does. This means sending someone else’s data to an external service, and no time saving is worth that. I do not paste client names or anything from which they could be reconstructed either.

For this post, I prepared a synthetic user story from scratch so that there would be no doubt about where it came from:

As a registered user, I want to log in to the dashboard
so that I can see my orders.

Acceptance criteria:
- the form has email and password fields and a "remember me" checkbox
- correct credentials take the user to /panel/
- incorrect credentials show one message without indicating
  which field is wrong
- after five failed attempts, the account is locked for 15 minutes
- "remember me" keeps the session after the browser is closed

Constraints: web application, "customer" role, Chromium and WebKit
browsers, only synthetic data from example.com domains.

The prompt I wrap around it is short and always the same:

You are a tester. Based on the user story and criteria below,
list test cases for the login flow.
Format: one line per case, in the form "condition -> expected result".
Group them by: happy path, validation, authorization, edge cases.
Do not propose code. Do not assume features absent from the criteria.

Two things in this prompt are deliberate. The instruction not to provide code, because I do not want a test file, only a list of sentences that cannot accidentally be pasted into the repository. And the instruction not to assume features absent from the criteria, which does not work but reduces the scale of the problem. More on that below.

How I review the output

The response to the prompt above, after combining the groups into a single list and completing my first pass, looks like this:

1. Correct email and correct password -> redirect to /panel/, order list is visible.
2. Correct email, wrong password -> one general message, no information about which field is wrong.
3. Email that does not exist in the system -> the same message as in item 2, without revealing whether the account exists.
4. Both fields empty, submission attempt -> form validation, no request to the server.
5. Email without an @ sign -> format message, password remains in the field.
6. Email with leading and trailing whitespace -> login works or the message is unambiguous; this depends on a decision absent from the criteria.
7. Four failed attempts, followed by the correct password -> login works, attempt counter resets to zero.
8. Five failed attempts -> account locked, message says the lock lasts 15 minutes.
9. Correct password during the lock -> login is still rejected, timer does not restart.
10. Login with a Google or Facebook account -> user returns to the dashboard without entering a password.
11. Performance test: 1000 concurrent logins, response time below 200 ms.
12. Click the #login-submit button, check the “Welcome back!” heading and the auth_token cookie.

I keep items one through nine. Two of them are worth more than all the others combined: number seven, meaning the attempt counter resets after a successful login, and number nine, meaning a correct password during the lock. These are exactly the two cases I forget when I write a list at nine in the morning after stand-up, and that is the real value of this tool. Number six is even more interesting because it is not a test case but a missing product decision. It goes to the user story’s author as a question, not into the set as a test.

There are three crossed-out items, each for a different reason, and those three reasons return in every session.

Number ten is a feature absent from the user story. The model added social login because such buttons sometimes appear on a typical login form. I explicitly asked it not to assume features outside the criteria, and that did not help, so I treat this instruction as a way to reduce the number of fabrications, not as a guarantee.

Number eleven is out of scope. There is no performance requirement, the number 1000 and the 200 ms threshold are pulled out of thin air, and adding them to the functional checklist leads to a discussion about a threshold that nobody set.

Number twelve is the most dangerous because it looks the most professional. The #login-submit selector, the “Welcome back!” heading, and the auth_token cookie name are three fabricated details delivered in the tone of someone who has seen the application. The model has not seen it. It does not know my DOM, it does not know the project’s copy, and it does not know what the session is called in this application. Anything that is a locator, interface text, or field name has to be checked against the application, always, without exception.

Separately, I maintain a second list that the model will not produce because it does not know what it does not know. For this user story, the missing cases were: logging in using only the keyboard and submitting with Enter, pasting the password from a password manager, returning with the “back” button after logging out, session expiration over time, and behavior after changing the password in another tab. Four of these five come not from the criteria but from the ways this application has broken in the past, and that knowledge is not present in the chat window.

My balance sheet after six weeks is this. The first draft of the list is created faster and is broader than the one I write manually. I remain the owner of the list, and half of the time I save on writing comes back as time spent reading and crossing things out.

From the list to the ticket, not to the commit

The most important decision in this entire process is where the list ends up.

It ends up in the checklist attached to the ticket and in the acceptance criteria, as text in Polish. It does not end up in .spec.ts or in a class with [Fact]. I still write the suite myself, manually, in Playwright on a pinned version, and it is the same 1.29.2 that my project has used since the beginning of January. The reason is not ambition. Test code created from a list of sentences has two kinds of content: structure, which is cheap, and locators with assertions, which are the entire value of the test. Item twelve above shows what the model does with the second category.

There is also a process reason. The list of cases in the ticket is visible to the user story’s author and the developer. The question in item six goes back to them before anyone writes a line of code, and that is a cheaper moment for this conversation than the test’s code review. A test created immediately would freeze my guess instead of prompting a question.

And there is the reason I have repeated since 2021. The place that determines whether a test suite means anything lies in the process, not in the tool. A faster first draft of the case list does not change who sets the entry and exit criteria, who looks at the overnight report, or what happens to a defect after it is reported. It only means that the first version of the checklist is ready before lunch.

Data remains a separate subject, and I am not changing anything about it. The conclusions from the post about the test data management strategy remain unchanged: data for assertions comes from a seeded factory or an approved file in the repository. The chat window can suggest an edge case, such as an email with a very long local part, but I do not copy the record it produces into a fixture. Two calls with the same input still do not have to produce the same result, and that is the same property I described a year and a half ago.

January pitfalls

Several things are simply true at this point, and I would rather write about them now than pretend I am working with a finished tool.

Availability. The free chat regularly responds with a capacity message, most often in the afternoon in my case. That is acceptable at my desk, when I can return twenty minutes later. It disqualifies this way of working anywhere something has to run at a specific time, and it is exactly the same class of risk I was escaping by putting stubs in containers.

No memory between sessions. Within one conversation, the context works and I can ask for more variants. A new conversation starts from zero: it does not know my conventions, does not remember that it fabricates locators, and does not know that login in this project has a fifteen-minute lock. I ended up with a file containing three paragraphs of context that I paste at the beginning, and it is less convenient than it sounds.

Confidence of tone. This is the least technical and the most costly pitfall. A fabricated locator is delivered just as firmly as a correct edge case, without a hint of hesitation and without a signal that can be caught visually during a quick read. That is why I separately cross out locators, interface text, field names, and all numbers absent from the criteria.

Nondeterminism. The same prompt and the same user story produce two different, partially overlapping lists in two conversations. That does not matter for drafting. For any measurement, such as “did case coverage increase,” it is useless, so I do not try to turn it into a metric.

Summary

Six weeks after the release, I have one use that I stick to: ChatGPT is my test-case scratchpad, open in the browser next to the ticket, and it shortens the time to the first version of the checklist. Along the way, it catches two or three variants that I would forget after stand-up, and I give it credit for that.

It is not a replacement for a tester because the entire difficult part of my work on this user story consisted of crossing out three items and adding five that the model did not know. It is not a suite generator because locators and assertions are the only parts of a test that really cost anything. It does not remove the isolation I built into the suite throughout 2021, and it does not remove the habit of reading the diff before a commit that I developed while working with Copilot’s grey text.

I want to know how many of the cases generated in January survive their first contact with the application and how many I delete during review. I do not need a new tool for that, just patience and the same tickets. I will return with a number when I have one.