Why a tester needs a language model
I have been reading about GPT-3 for a year, and during that year I have mainly seen two things: demos in which the model writes a poem, and demos in which the model pretends to be a conversation partner. Neither one is my professional problem. My professional problem is called “where do I get twenty sensible user records so that the list screen does not look like test1, test2, test3“.
That is why, when I got access to the API, I did not move toward conversation. I moved toward the Completions endpoint, which means text goes in and text comes out. I am interested in one question: can I use it to generate a fixture that I put into the test repository, and is this route cheaper than writing a data factory by hand?
I want to set expectations right away, because otherwise this post would sound like an advertisement. This is an experiment. I am not switching my regression suite to this, and I am not encouraging anyone else to switch theirs.
What the API is in June 2021
OpenAI announced the API on June 11, 2020, exactly one year ago. Since then, access has been by invitation: you fill in a form and wait for your turn. I am saying this explicitly at the beginning because if someone wants to reproduce this post line by line today, the first step is not pip install, but joining the waitlist. I had to do my waiting too.
The interface itself is surprisingly simple, and that is its greatest strength. There is no form with fields, and there is no set of separate methods for separate tasks. There is one operation: I send text and receive its continuation. I describe the task in the same text in which I ask for the result.
There are four models, and they differ in capability and price: ada, babbage, curie, and davinci. ada is the fastest and cheapest, while davinci is the most capable and most expensive. I am sticking with davinci in this experiment because I want to see the upper limit of what is possible today. If this were ever to enter regular use, the first thing to check would be whether curie is sufficient.
The endpoint in question looks like this:
POST https://api.openai.com/v1/engines/davinci/completionsThe engine name is part of the address. Changing davinci to curie means changing one word in the URL, and that is essentially the entire mechanism for switching models.
One concept is worth explaining because it will return later. The work described in the May 2020 paper “Language Models are Few-Shot Learners” shows that a model of this class learns a task from a few examples provided directly in the query text. I do not fine-tune the model and I do not have my own training set. I show it three or eight examples and ask for the next element in the same series. This is exactly the mode that interests me for test data.
Setup: key, openai==0.7.0 client, first call
I keep the key in an environment variable and only there:
$ export OPENAI_API_KEY="sk-..."I follow three rules here. The key never goes into the repository, not even into a configuration file “just for a moment”. The key never goes into logs, so I do not use print(api_key) or print request headers while debugging. The key does not go into CI variables because, in this experiment, CI does not call the API at all, as I explain below.
The official Python client is on PyPI. I pin the version:
$ python --version
Python 3.9.5
$ pip install openai==0.7.00.7.0 was published on June 11, four days ago. Before that, I used 0.6.4 from May. I pin the exact version for the same reason I pin image tags: I want a colleague reproducing my script a month from now to get the same library, not whichever one happens to be released by then.
The simplest call that makes sense:
import os
import openai
openai.api_key = os.environ["OPENAI_API_KEY"]
response = openai.Completion.create(
engine="davinci",
prompt="List of three cities in Poland:\n1.",
max_tokens=40,
temperature=0.7,
stop=["\n\n"],
)
print(response["choices"][0]["text"])There are four parameters that I set deliberately in every call from this point on.
engine selects the model. max_tokens is a hard limit on the response length. It is not a character or word limit, but a token limit, meaning the chunks of text into which the model divides the input and output. I always set it because without it the model can keep going longer than I need, and I pay for what it generates.
temperature controls randomness. At 1.0, I get creativity that I usually do not want in test data. At 0, the responses are the most conservative and most repeatable, although I cannot call that a guarantee of determinism because it is not a contract, only an observation from several dozen calls.
stop is the sequence after which the model should stop. It is the most underestimated parameter in the entire API. Without it, after generating my JSON object, the model keeps going and adds a comment, another heading, or a sentence about how useful test data is. With stop=["\n\n"], it ends where I want it to.
Here is a .NET variant, because I work with .NET 5 in client projects. There is no official SDK for C#, so I go directly through HttpClient:
using var http = new HttpClient();
http.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(
"Bearer",
Environment.GetEnvironmentVariable("OPENAI_API_KEY"));
var payload = new
{
prompt = prompt,
max_tokens = 160,
temperature = 0.2,
stop = new[] { "\n\n" }
};
var body = new StringContent(
JsonSerializer.Serialize(payload),
Encoding.UTF8,
"application/json");
var response = await http.PostAsync(
"https://api.openai.com/v1/engines/davinci/completions",
body);
response.EnsureSuccessStatusCode();This is a regular POST with an Authorization: Bearer header. The field names in the request body are the same as the Python client arguments, so one explains the other. I draw attention to EnsureSuccessStatusCode and to the fact that I log neither the header nor the entire request object. A key in a build log is a security incident, not an inconvenience.
A few-shot prompt for a JSON fixture
Now for the core of it. I do not ask the model to “generate test data” because then I get an essay. I ask it to add another element to a series that I started myself. The prompt is a list of complete objects, one per line, cut off halfway through the next one:
List of synthetic test users for a store, one JSON object per line.
{"email": "anna.kowalska@example.com", "locale": "pl-PL", "displayName": "Anna Kowalska", "newsletter": true}
{"email": "jan.novak@example.org", "locale": "cs-CZ", "displayName": "Jan Novak", "newsletter": false}
{"email": "boundary.case.with.a.very.long.local.part@example.com", "locale": "en-GB", "displayName": "A", "newsletter": false}
{"email":Four decisions here are deliberate.
The addresses use the example.com and example.org domains, which are reserved for documentation. None of this will send anyone an email if the test environment configuration turns out to have a hole in it.
The format is one object per line, not an array. An array requires the model to close the bracket and keep track of commas throughout the response, and that is exactly the class of error that breaks my parsing. Line by line is more robust and easier to cut off.
The third example is ugly, and it is meant to be. It has a very long local part of the address and a one-character displayName. I show the model that this series includes edge cases because otherwise I will get twenty variations of Anna Kowalska.
The prompt ends at {"email":. This leaves the model no room for an introduction such as “Of course, here is another user”. The only sensible continuation of the text is the rest of the object.
The response does not go straight into the repository. It goes through validation first:
import json
REQUIRED = {"email", "locale", "displayName", "newsletter"}
def parse_line(raw: str) -> dict:
record = json.loads(raw)
missing = REQUIRED - record.keys()
if missing:
raise ValueError(f"missing fields: {sorted(missing)}")
extra = record.keys() - REQUIRED
if extra:
raise ValueError(f"fields outside the contract: {sorted(extra)}")
if "@" not in record["email"] or not record["email"].endswith(
("example.com", "example.org")
):
raise ValueError(f"email outside documentation domains: {record['email']}")
if not isinstance(record["newsletter"], bool):
raise ValueError("newsletter is not a bool")
return recordThis is not defensiveness just in case. Each of these four assertions triggered in practice during my first hour of experimenting. In a .NET project, I do the same thing by deserializing into a specific class with JsonSerializer and using a setting that rejects unknown fields instead of silently ignoring them.
What works and what breaks tests
I will start with what went well, because some things did.
The model really does follow the shape of the series. I get valid objects with sensible combinations of name, address, and locale, including ones I would not have thought of myself: surnames with diacritics that do not exist in the Polish alphabet and locale values for regions I had forgotten about. For filling a list on a screen, visual review, and a presentation for the team, this is better material than test1, test2, test3, and it is created in several seconds instead of half an hour.
Now for the list of things that break a test unless they are accounted for in advance.
Non-determinism. Two identical calls produce two different results. This is not a defect, but a feature. If a test asserts the content of a record and that record is created during the run, the test is flaky by definition, and no temperature=0 will save me from that.
Hallucinated format. The model can add a field that did not appear in the examples, such as phone or age, because it fits a user record. It can write pl_PL with an underscore instead of pl-PL. It can insert "newsletter": "true" as a string. All three pass through json.loads without blinking. The validation above catches the first and the third, but pl_PL is a perfectly good string as far as it is concerned, so locale needs its own check unless I want it to fail at an assertion or, worse, in the application layer.
Cost. Billing is based on tokens, and all traffic counts, meaning my prompt together with the response. A few-shot prompt with eight examples is sent with every call. I do not give rates here because pricing is something that must be checked at the source on the day it is used. I give the conclusion: generating data in a loop on every test suite run means paying on each run for something I am about to discard anyway.
Limits and availability. This is an external service over HTTP, with a request limit and the possibility of being unavailable. Letting it into a test run means that a red result may no longer mean “the application is broken”, but “someone else’s API had a bad day”. This is the same problem I was escaping by mocking a partner API with WireMock, and it would be quite funny to let it back in through the same door I had just shown it out of.
Model biases. In the API announcement, OpenAI explicitly writes about the limitations and biases of this family of models. When generating human first names, surnames, and demographic data, this is not an academic observation. A generated user set can be less diverse than it appears at first glance, so I do not treat it as a representative sample of the population.
There is one simple conclusion from all of this. I call the API once, manually, at my desk. The result goes through validation, then a person reads it, and the approved records end up in a file in the repository:
fixtures/
users.approved.jsonThe test reads the file. The test does not know the API key and has no right to access the network. The file goes through review like any other code. This way, the result of the run depends on the contents of the repository, not on the randomness of the model or someone else’s uptime.
The last rule, and the most important one in this entire post: do not paste production data into the prompt. Not one real address, not one real customer name, and not a fragment of a database dump. The temptation is real because “the model will pick up our format” sounds reasonable. That means sending someone else’s personal data to an external service, and no format is worth that. The examples in my prompt were invented from scratch and placed in documentation domains.
When not to use GPT-3
After two evenings of experimenting, I have a fairly specific list of situations in which this is a bad choice.
When I need a thousand records, not twenty. A seeded factory generates them locally, in milliseconds, for free, and identically on every run. In .NET, I use Bogus for this; in Python, a simple function using random with a set seed works just as well. One hundred records from a factory take one line of a loop. One hundred records from the API mean a dozen or so paid calls, each one paying for the full prompt again, and one hundred opportunities to hallucinate the format.
When the data must be repeatable. This is the same idea as pinning an image tag in Docker for QA. The dataset on which an assertion relies must be the same thing in January and March. A seed gives me that; the model does not.
When compliance with business rules is required. A valid bank account number checksum, a postal code matching the country, or consistency between locale and the address format. The model will generate something that looks correct, but it does not calculate checksums. This calls for a generator that knows the rule, not one that knows the shape.
I also want to clear up one misunderstanding separately because I have already heard it. When I described starting a database from a test in April, I was solving an infrastructure problem: where to get a clean MSSQL instance and how to clean it up. That is a completely different layer from today’s topic. Testcontainers gives me an empty database. It has no opinion about what should be in a record. GPT-3 has an opinion about the content of a record and no opinion at all about where that record will live. These two things neither replace nor compete with each other.
Summary
I am left with this division. For large amounts of repeatable data on which assertions depend, I use a seeded factory. For small, “human-looking” sets that a person will review anyway, I allow GPT-3 as a tool used at my desk, with the result approved and added to the repository. For the infrastructure underneath that data, I have containers.
Is this my new standard for test data? No. This is an experiment that produced an interesting result and cost me two evenings and a small token bill. Access requires joining a waitlist, the result is non-deterministic, and every generated record must still go through validation and someone’s eyes before it ends up in the repository. Those are a lot of conditions for something that was supposed to save time.
I am left, however, with a question that is broader than the tool itself and has been on my mind for some time: where does the data in my test suites come from in the first place, who owns it, and what happens to it between runs? A generator is only one possible source, alongside a factory, a snapshot, and an anonymized dump. I want to organize this into one coherent strategy in the autumn and describe it separately.
Before I add another tool to the set, however, I return to the question I ask myself every time: where does this fit into the entire test process? Better-looking data does not fix the process. It only makes what is not working in it visible sooner.