GPT-4 in QA - the day after launch, Plus and the API waitlist

Yesterday GPT-4, today my spec

It is 15 March. Yesterday evening, OpenAI announced GPT-4, and since then I have mainly seen two things online: tables of bar exam results and claims that this profession will no longer exist. Neither is the question I have for this model.

My question is narrower and fits into one sentence: does the model switch in the ChatGPT Plus interface change anything in the two activities I actually perform at my desk? The first is drafting a list of test cases from a supplied specification. The second is reading that specification for gaps before anyone starts coding.

I am writing this post the day after the announcement, and I want that to be clear. I have not completed a sprint with the new model, I do not have a chart covering three months, and I do not have an opinion on what this looks like in a team of ten people. I have one evening, one synthetic specification run through twice, and eight differences I can count. That is all I have, and that is all I will write about.

In my 2022 summary, I wrote down a sentence that I am returning to today: artificial intelligence entered my editor as a prediction of the next line and did not enter anywhere else. Three months later, that sentence is still true because I have exactly the same thing in my editor as I did in December: gray text in front of the cursor, Tab, and Esc. What I am writing about today happens in a browser window, in a separate tab, next to my IDE. This boundary matters to me, and I will return to it at the end.

What is available on 15 March

Before discussing any use cases, I am establishing the facts because everything is shifting this week, and six months from now no one will remember what was behind a gate and what was not.

In ChatGPT Plus, the subscription that costs 20 dollars per month, I have a new GPT-4 option in the model dropdown. I select it per conversation. There is a limit on the number of messages within a time window, and OpenAI immediately said it would adjust that limit depending on load, so I do not treat any specific number as permanent. The practical consequence is that I cannot use this model all day in a continuous back-and-forth. I need to have the input prepared.

I do not have gpt-4 in the API. There is a waitlist, and I filled in the form like everyone else. OpenAI published the model names and pricing in advance: the gpt-4-0314 snapshot, a context window of 8192 tokens in the standard variant and a separate variant with 32768 tokens, separate billing for input and output tokens, and rates clearly higher than what I have in my basket today. I am recording this as information, not as something I have in a script. Until I get access, any code of mine calling gpt-4 would be fiction.

What I have had in the API for two weeks is gpt-3.5-turbo. Chat Completions was released on 1 March, with the gpt-3.5-turbo-0301 snapshot, a 4096-token window, and a price of 0.002 dollars per thousand tokens. This is ten times cheaper than the text-davinci-003 line I used before, and that is a change that genuinely moves the bill for experiments from “I need to think about this” to “I will not notice it on my statement.” The request structure is also new: instead of one block of text, I send a list of messages with roles, including a system role where I can set the response format once for the entire conversation.

The Completions endpoint with davinci in the address, which powered my 2021 test data experiment, still works. I am not rewriting that post or changing its conclusions because they were not about the model, but about nondeterminism and where the approved fixture lives.

I am noting two things without making them the subject of this post. GPT-4 accepts images as input, but that part is in a limited alpha with one partner, so it is not a tool I use today, and I will not write a tutorial about it. On the same day, Anthropic announced its Claude model, and I am leaving it at one sentence because I have neither access to it nor a single run on my own data, while comparing other people’s benchmark tables is not the content I want to produce.

Where GPT-4 helps in QA

I made the test as simple as I could so that the comparison would not become a story about impressions. The same specification, the same prompt, two conversations in the same interface, one using the default model and one using GPT-4.

The specification is synthetic, and I wrote it myself for this post. It does not contain a single sentence from any project I work on, and no client will be named here. I invented a “discount code in the basket” feature for a store that does not exist: nine acceptance criteria written in the style in which they arrive in tickets, plus four cross-cutting constraints embedded in the description paragraphs rather than in a list. Those four are one code per order, the minimum basket value calculated before the discount, the currency matching the store’s market, and the expiration date using the store’s time zone rather than the user’s. I hid the constraints deliberately because this is exactly where they sit in a real specification and exactly where they disappear from a draft.

The prompt was identical in both conversations: the role of a tester, a request for a list of cases in a table with columns for precondition, steps, and expected result, an explicit instruction to mark places where the specification is silent, and a ban on inventing interface element names. Then there were four iterations in both conversations: add boundary cases for the amount, add negative cases for an expired code, shorten each step to one sentence, and add a priority column.

I counted the result manually. This is one run on one specification, so these are not statistics, only notes from an evening.

The two columns in this table say two different things, and I do not want them to merge into one sentence about the new model being better.

The first thing is real: longer input stops getting lost. The default model behaved as if it read my specification from the top and forgot the beginning along the way. The four constraints embedded in the description paragraphs were invisible to it; it caught one, the most obvious one. GPT-4 extracted three out of four, and for two of them it independently added a sentence that the specification does not say what happens when the basket drops below the minimum after a product is removed. This is exactly the point you take to an analyst and exactly the point the default model did not raise once. Four marked gaps versus zero is the most important row in the entire table for me, more important than the number of cases, because I can produce a number of cases myself, and that number is no achievement.

The second thing is also real: the format holds up longer through iterations. With the default model, after two revisions I got a table with a different number of columns and cases that had previously been there but now disappeared without a word. This is the class of behavior where all the time savings evaporate because I start comparing version three with version one. With the same four revisions, GPT-4 returned a fifth version with the same columns and without entire rows disappearing. A lost constraint appeared only on the fourth iteration, which means it happened later, not that it stopped happening.

Now for the thing the new model did not change, because this is the point of this post. Fabrication did not disappear. Three invented interface element names dropped to two, even though my prompt explicitly prohibited them. The model still produced #discount-code-input and data-testid="apply-voucher" on its own. They do not exist anywhere because this store does not exist, and the specification did not contain a single selector. There was one invented endpoint in both cases: POST /api/cart/voucher, with a field that my text never mentioned. In one of them, GPT-4 added a sentence that the name was an example and required confirmation. I appreciate that, and it is still the same problem because in the fifth version of the table, after the steps were shortened, that caveat did not return, but the selector remained.

The practical conclusion is the same one I wrote down in June when Copilot left technical preview, only at a different layer. The model is good at shape and weak at specifics that depend on a reality it cannot see. The shape of a case list, division into positive and negative scenarios, naming an equivalence class, a reminder about something I missed - yes. A selector, an endpoint address, a field name, a boundary value derived from a business rule - no, and I check every one of those four categories. In my work, selectors are not created in a chat anyway: they are written with the page open and the runner running, in my case Playwright in 1.31.2, and there is no shortcut for that.

There is also a cost that is easy not to mention. Preparing the input for this comparison took me more time than the two conversations themselves. The specification had to be ready, the prompt had to be the same, and the result had to be read in full to catch the silent loss between the second and third versions. A model that gives me twenty-one cases in thirty seconds does not shorten the reading. It moves my time from writing to reviewing, and that is the entire transaction I am making here.

Where I am staying with 3.5

I am not switching to the new model everywhere, and I have three reasons, all mundane.

The message limit in Plus is a real barrier for work that consists of twenty short questions in a row. When I need a draft from which I will discard half anyway, such as five names for a parameterized test or a quick reminder of the equivalence class for an amount field, I choose the default model without hesitation. It responds faster, and the quality I need is “something to start from.”

API pricing points in the same direction. Everything I have automated at my desk runs on gpt-3.5-turbo at 0.002 dollars per thousand tokens, and at the rates announced for gpt-4, there is no reason to move tasks there when I will still read the result sentence by sentence. If I got API access today, the first thing I would do would not be rewriting scripts, but selecting the one task where longer context actually changes something.

The third reason is about me, not the model. A short task with short input is exactly the case where the difference from my table does not occur. The entire advantage I saw yesterday concerned long input and many rules at once. With three lines of context, both models give me the same thing, and paying for the better one is simply wasting the limit.

What I am not doing this week

The list of things I am deliberately not doing is more important this week than the list of what I did.

I am not connecting any model to CI, neither gpt-3.5-turbo, which I can access, nor gpt-4, which I cannot access. The conclusion from 2021 has not changed by a comma: two identical calls do not have to return the same result, so a test whose result depends on the model’s response is flaky by definition, and a red run starts to mean “someone else’s API had a bad day” instead of “the application is broken.” The model can be a tool that helps me write a case. It is not a step in the pipeline that executes that case.

I do not send someone else’s specification to the chat. That is why this post contains a specification I invented, and it will be the same in every subsequent post about these tools. Pasting a project ticket into an external service is a decision for the owner of that ticket, not mine, and it is not done quietly to see whether the model can handle it. The same rule I wrote down when generating data applies: nothing from production, not one real name, not one fragment of a dump.

I am not changing anything in the editor. For me, Copilot still does the same thing it did in December, which is suggest the next line, and there is nowhere in it where I can ask it anything. The conversation about the specification sits in a browser tab, the code is created in the IDE, and today these two things have no technical connection. I am not forcing such a connection into existence or planning work on the assumption that one will appear soon.

I am also not rewriting my earlier posts about these tools. The 2021 post describes the Completions endpoint from June 2021, and it stays that way. The December summary describes the year in which gray text appeared in my editor, and it stays that way. Adding paragraphs with today’s knowledge to the archive only makes it impossible a year from now to reconstruct what was known when.

Summary

After one evening, I have a conclusion that fits into one sentence: GPT-4 in Plus is clearly a better drafting tool for a long specification, and that is the whole change.

I am breaking this down into three sentences that I want to be able to read a year from now. Longer and more tangled input stops getting lost halfway through, which makes a real difference when reading a specification for gaps because, for the first time, the model independently pointed out four places where something was missing. The response format holds up longer through iterations, so less time goes into comparing table versions. Fabricating selectors, endpoints, and boundary values has not disappeared; the model only does it less often and admits it more gracefully, so I continue to check all four categories exactly as I did in December.

What does not follow from this? It does not mean that I have a new tool in the pipeline, because I do not. It does not mean that I have API access, because I have a waitlist form. It does not mean that anyone is no longer needed, because a model that invents a data-testid for a store that does not exist is not a candidate to own the regression suite.

I remain the owner of the test cases, and that is not false modesty. I put my name to the list the team receives, and you can only put your name to something you have read. The new model shortened my path to the first version of that list and did not shorten the path from the first version to the version I send by even a minute. The latter has always been more expensive, and it still is.

In three or four weeks, I will know whether the advantage from the table holds for specifications I did not write myself or only for the one I prepared for the comparison. Until then, I return to the question I ask about every new tool and which has not changed since 2019: where does it fit in the entire testing process? A better draft of a case list does not fix a process where no one reads that list before coding. It only means that the list is ready earlier and that it becomes clear sooner whether anyone is waiting for it.