Why a second post about chat
It is February 15, and I have had several weeks during which I drafted test cases with chat in January and drew one conclusion from it: the model can write a list that looks like one from a ticket, and it can write it in thirty seconds. That was a conclusion about the tool. Today’s post is about something else, because the problem I was left with after January has nothing to do with whether chat can write cases.
The problem is this: I asked almost the same question twice and got answers of two different qualities. Once, I got a table that I pasted into a ticket after one correction. Once, I got an essay on how important it is to test forms, with five cases at the end, three of which covered fields that do not exist in this form. The difference was not in the model, because the model was the same. The difference was in what I wrote.
That is why today I am not writing a second introduction to ChatGPT, and I am not writing about any new version. I am writing about a prompt as something that you put together once, save in a file, and then only replace the context in it. The entire post has one thesis: in February 2023, the quality of a list of cases from chat depends less on whether I have purchased the paid plan and more on whether my message includes a role, hard constraints, an output format that can be pasted somewhere other than the browser window, and whether I made a second round for gaps instead of taking the first dump.
One housekeeping note at the start, because without it the rest of the post would be dishonest. Everything below happens in a conversation window, in a browser, manually, and ends with copying text into a ticket. There is not a single line of code here that calls anything. This is not a post about plugging the model into a pipeline, and I will not have such a post in February because I have neither an interface for it nor a reason to do it.
The four building blocks of a prompt
The prompt I use today has four parts, and I make sure that each is written separately, even when that makes the message long. Their presence matters more than their order.
The first building block is the role, and this is where I most often see other people’s mistakes. The role should be narrow and boring. “You are a QA engineer who writes test cases for manual execution” works better for me than “you are a senior test architect with twenty years of experience,” because the second sentence asks the model for tone, not content, and in return I get a paragraph about quality strategy instead of a row in a table. The more the role sounds like a job title from LinkedIn, the more prose I get.
The second building block is context, and here I follow a rule to which I make no exceptions: the context is synthetic. I do not paste client names, I do not paste real environment URLs, and I do not paste personal data from the test database. This entire post is based on a fictional cart from a fictional store, and this is not caution for caution’s sake. A browser conversation is on someone else’s side, and I have no control over it, so only things I could publish on the blog go into it. As a consequence, I have to describe the screen in words, and that is work the chat will not do for me.
The third building block is constraints, and it determines how many corrections I will make later. A model without constraints fills gaps with the most probable content, not the true content, and this is the same observation I wrote down last March about suggestions in the editor: systematically probable instead of correct. That is why I have three sentences that go into every prompt: do not invent fields I have not provided; do not invent endpoint names or response codes; if you do not know something, list it as a question, do not guess. The last instruction is the most valuable of the three, because it turns a hallucination into a list of things I need to ask the team about myself.
The fourth building block is the output format, and the test for it is trivial: can it be pasted into a ticket without rewriting it? If I have to split paragraphs into steps manually, then the prompt was wrong, not the answer. I ask for a Markdown table with fixed columns and a separate section for questions.
Put together, my template today looks like this:
Role: you are a QA engineer who writes test cases for manual execution.
Do not write an introduction or summary.
Context: the "order summary" step in an online store's cart.
The form has exactly these fields: email, first name, last name, street and number,
postal code, city, delivery choice (courier or parcel locker),
payment choice (card, bank transfer, cash on delivery), terms acceptance (checkbox).
The "Pay" button is disabled until the required fields are completed.
Web application, Chromium and WebKit browsers, desktop and mobile versions.
Constraints:
- do not invent fields that are not on the list above,
- do not invent endpoint names, response codes, or validation messages,
- if you do not know something, add it to the "questions" section and do not guess,
- no case may depend on the result of the previous case,
- no more than 12 cases in this round.
Format: Markdown table, columns
id | precondition | steps | expected result.
After the table, include a "questions" section as a bulleted list.The things that are deliberately absent from this template are as important to me as those that are present. There is no request for priority, because priority is not a property of the form but a property of risk in a specific project, and the model has no data about it. There is no request for ready-made test code, because then I get code for invented locators, and the review takes longer than writing it myself. There is no number such as “generate 50 cases,” because the twelfth row is usually already a variation of the third, and by the fiftieth I am reading nothing but duplicates.
Iteration instead of one shot
The first answer to the prompt above is always similar in nature for me. I get coverage of the happy path, one case for each delivery and payment option, two or three cases for an empty email and an invalid email format, and a questions section in which the model sensibly asks whether the parcel locker requires selecting a location on a map. This is good input material and a bad final list.
It is bad because it is missing exactly what usually breaks checkout in production. That is why the second message is a permanent part of my process, not a failure of the first prompt:
This list covers the happy path and email validation.
Add only new rows, in the same format, for:
- required fields left empty, each separately,
- a postal code in an invalid format and with an invalid length,
- no acceptance of the terms when the rest is completed correctly,
- a value at the maximum allowed field length and one character longer,
- two simultaneous purchases of the last unit of the same product,
- returning with the "back" button after submitting the form,
- refreshing the page while filling it in.
Do not repeat cases you have already listed. If any point
requires backend knowledge you do not have, add it to the "questions" section.The value of this message does not lie in the model being unable to come up with these things itself. It lies in the fact that I list the gap categories myself, from memory, and they are the same every time: negative cases, empty values, boundaries, concurrency, navigation, and browser state. I have them saved next to the template as a simple checklist, and I do not pretend that artificial intelligence provides them to me. Chat fills the categories with content that fits the described screen and does it faster than I could click through it myself, and that is not the same skill.
I follow two habits around the second round. I do not reset the conversation without a reason, because the entire context of the form is already above, and a new chat means pasting it again with the risk that I simplify something in the process. I reset only when the model has anchored itself to a wrong assumption and stubbornly returns to it in every subsequent answer despite a correction; then it is cheaper to start from a clean template with one sentence added that rules out that assumption. The second habit: I read the questions section first, before the table. It is the only place where the conversation tells me something about my own description of the system rather than about the form.
And one sentence I have to state directly, because without it this whole technique sounds better than it works. The third and fourth rounds usually add nothing. After the second round, the curve flattens, the rows begin to repeat themselves in different words, and I begin to delete more than I add. I stop after two rounds and continue working in tickets.
Plus, free, and another chat alongside them
Since February 1, there has been a paid ChatGPT Plus plan for 20 dollars per month, and since February 10, it has been available outside the United States, so for five days I have actually been able to buy it from Poland. I will write exactly as much about it as it is worth in the context of this post.
Plus buys availability: access during hours when the free version returns a capacity message, faster responses, and priority access to new features. This is a noticeable difference when I want to work in the middle of the day. It does not, however, buy a single thing I wrote about above. The same prompt without constraints or a format produces an equally rambling answer on the paid plan, only I get it sooner. The division of labor does not change either: what remains to be reviewed remains to be reviewed on both plans.
For completeness, because it cannot be ignored this week: on February 7, Microsoft showed the new Bing and Edge with chat connected to search, meaning a second chat with a different profile appeared alongside my window. I note it as a calendar fact and do not mix it into this post, because generating a list of cases from a description of a form that I provide myself is not a task where access to fresh search results changes anything.
What remains on my side
Three things from this process have not moved to the model, and I do not see a path by which they would move this year.
Priority and which items from the list are included in the scope at all. The model gave me twenty-something rows after two rounds, and all of them are formally correct. Which ones we execute before releasing the change and which land in the “someday” bucket depend on where this store actually loses money and what has broken in it recently. Chat does not have this data, and I have no way to provide it without breaking the rule about synthetic context. This is the same boundary that all the tools described on this blog run into: the decision lives in the process, not in the tool.
Test data. The table says “enter a postal code of an invalid length,” and that is an instruction, not data. Where we get an account with an order history, who clears the cart after a failed payment, and whether the dataset can be recreated twice with the same result are questions answered by the test data management strategy, and nothing in February has changed it. I also have my own older experiment with generating the data itself through the HTTP interface in Completions, and its conclusion still holds: two calls with the same input do not have to produce the same result, so this is a tool for the ideation stage, not a source of truth in a CI run. A conversation in the browser is worse in this respect, not better, because I do not even have parameters to pin there.
Whether a case is worth automating. A row in the table costs one click to copy. The same row rewritten as a test in Playwright, for me today on 1.30.0 from January 25, costs locators, data, time at the gate, and maintenance for the next two years. The ease of generating the list pushes the temptation in the wrong direction here, and this is the most real risk of this entire technique for me: suddenly I have three times as many candidates for automation with the same maintenance budget. I described the division between what goes into a pull request and what goes into a nightly run in September, and it remains unchanged, regardless of where the case text came from.
On top of that, there is a contrast I repeat yet again because it keeps getting blurred in conversations. Copilot in the editor is gray text in front of the cursor, without a window for asking questions, and its result is my code. A conversation in the browser is a separate tool whose result is text for a ticket. Two different places in my day, two different kinds of risk, and neither replaces the other.
Summary
Prompt engineering sounds like a field, but for me in February 2023 it is the simple discipline of writing a single message: a narrow role, synthetic context written out manually, constraints that prohibit guessing, and a format ready to paste. Plus does not change that, it only shortens the queue. A second round based on gap categories gives me more than any new model, because the categories come from my head anyway.
After a month of working this way, I can describe the effect only qualitatively, and I will not pretend to have a number: the time between “I have a described screen” and “I have a list of cases in a ticket” has decreased noticeably, while the time needed to decide which of those cases mean anything has not decreased by even a minute. In the 2022 summary, I wrote that artificial intelligence entered my editor as prediction of the next line and nowhere else. After February, I add one sentence to that: it also entered the browser window as a tool for drafting text that I then review myself, and it still has not entered any file on which a merge to main depends.
The template stays in my notes repository, next to the list of gap categories. If, in a few months, a way appears to call the same prompt other than with my fingers in a browser, I will still start with the same four building blocks, because they do not belong to any interface.