All posts

Synthetic test data from an LLM - gpt-3.5-turbo in June 2023
Synthetic test data from an LLM - gpt-3.5-turbo in June 2023

Polski

Synthetic test data from an LLM - gpt-3.5-turbo in June 2023

June 2023: Chat Completions for fixture JSON. Schema review, no GPT-4 API, no PII.

The same problem as two years ago, a different endpoint

It is 15 June 2023. Exactly two years ago, in June 2021, I described an experiment with the Completions endpoint and the davinci model here: a few-shot prompt, a dozen or so user records, a result approved by hand and committed to the repository. I am not rewriting that post or changing its conclusions, because they were not about the model. They were about nondeterminism and where the approved fixture lives.

The problem I have today is the same as it was then. I need a few dozen orders so that the list screen and monthly report do not look like test1, test2, test3. Two technical details have changed. The endpoint is now /v1/chat/completions, and the model I can access is gpt-3.5-turbo, available through the API since 1 March. Instead of one block of text, I send a list of messages with roles, including a system role where I set the response format once for the entire conversation. This is a real improvement for data because “return JSON lines only” is no longer a sentence the model loses after the third paragraph of the prompt.

One sentence about what is not here, so there is no doubt when reading the code. The gpt-4 API is still behind a waitlist, just as it was in March when I compared both models in the Plus interface. Everything I call from a script below runs on gpt-3.5-turbo. If I put code calling gpt-4 in this post, it would be fiction, not a tutorial.

What matters most, however, has not changed at all: the layer below. A language model replaces neither a seeded factory nor a database reset. The four layers of test data I laid out in September 2021 in my .NET data strategy remain exactly where they were, and the LLM generator does not occupy any of them. It is a step before them: it produces a candidate row that reaches the file used by the factory to seed data only after review.

Data contract first, prompt second

Rule number one is that the model does not receive an instruction to “invent orders.” It receives a shape. When I ask an open-ended question, I get an essay with fields that were never mentioned and values that look real. When I ask it to add items to a series I have defined myself, I get something that can be parsed.

I invented the contract for this post from scratch, and it does not contain a single field from any project I work on. A synthetic Order, with exactly three fields:

public sealed record Order(
    string OrderNumber,   // ORD-2023-NNNNNN, unique
    string CustomerEmail, // must exist in the customers table
    decimal TotalGross);  // 2 digits after the decimal point, 10.00 - 4999.99

Three fields are enough to show all three classes of problems covered in this post: uniqueness, a foreign key, and number formatting. CustomerEmail is deliberately a foreign key here, not decoration. In the test database, customers come from a seeded factory, so the set of allowed addresses is known in advance and closed.

I assemble the prompt from the same building blocks I described in February when discussing the structure of a prompt for test cases: role, constraints, output format, examples. There is one important difference. There, I read the result with my own eyes. Here, the result goes to a parser, so the output format stops being a convenience and becomes a contract.

The system message is short, and that is its entire value:

Return JSON lines only, one object per line.
No introduction, no comments, no Markdown block, and no summary at the end.

The user message is long, and this is where the contract lives:

You generate synthetic test data. Record schema, exactly three fields:
- orderNumber: string, pattern ORD-2023-NNNNNN (six digits), unique across the entire response
- customerEmail: string, exactly one of the values from the allowed list below
- totalGross: number, period as separator, exactly two digits after the decimal point, from 10.00 to 4999.99

Allowed addresses (do not invent others):
anna.kowalska@example.com
jan.novak@example.org
b@example.com

Restrictions: no other fields, no other domains, no text outside the JSON lines.

Examples:
{"orderNumber": "ORD-2023-000101", "customerEmail": "anna.kowalska@example.com", "totalGross": 249.90}
{"orderNumber": "ORD-2023-000102", "customerEmail": "b@example.com", "totalGross": 10.00}

Add another 28 lines to the same series.

Four decisions here are deliberate, and they are exactly the same decisions I made in 2021.

The addresses use the example.com and example.org domains, which are reserved for documentation. None of this will send email to anyone if the test environment configuration turns out to have a hole in it. The third example is ugly by design, because without it I get twenty variants of the same Anna. The format is one object per line rather than an array, because an array requires the model to keep track of commas and the closing bracket across the entire response. The number of records is explicit in the prompt so that the result can be counted before parsing.

And the rule that matters more than the rest of this post combined: nothing from production goes into the prompt. Not one real customer address, not one name from the database, not a fragment of a dump, not a screenshot from an analytics tool. The temptation is real because “the model will pick up our format” sounds reasonable and does work. It also means sending someone else’s personal data to an external service, and no format is worth that. The list of allowed addresses in my prompt comes from a factory, not a database.

I keep the API key in the OPENAI_API_KEY environment variable and nowhere else. It is not in the repository, not in a log, and not in CI variables because, in this setup, CI does not call the API at all.

From response to seed

The model’s response is not a fixture. It is input for review, and the first thing it meets is a parser that is not having a good day.

private static readonly HashSet<string> Contract =
    new() { "orderNumber", "customerEmail", "totalGross" };

private static readonly HashSet<string> AllowedEmails = new()
{
    "anna.kowalska@example.com",
    "jan.novak@example.org",
    "b@example.com"
};

private static readonly Regex NumberPattern = new(@"^ORD-2023-\d{6}$");

public static Order Parse(string line, ISet<string> alreadyApproved)
{
    var raw = JsonSerializer.Deserialize<Dictionary<string, JsonElement>>(line)
        ?? throw new FormatException("empty object");

    var extra = raw.Keys.Except(Contract).ToArray();
    if (extra.Length > 0)
        throw new FormatException($"fields outside the contract: {string.Join(", ", extra)}");

    var missing = Contract.Except(raw.Keys).ToArray();
    if (missing.Length > 0)
        throw new FormatException($"missing fields: {string.Join(", ", missing)}");

    var number = raw["orderNumber"].GetString() ?? "";
    if (!NumberPattern.IsMatch(number))
        throw new FormatException($"orderNumber outside the pattern: {number}");
    if (!alreadyApproved.Add(number))
        throw new FormatException($"orderNumber: duplicate {number}");

    var email = raw["customerEmail"].GetString() ?? "";
    if (!AllowedEmails.Contains(email))
        throw new FormatException($"customerEmail outside the list: {email}");

    if (raw["totalGross"].ValueKind != JsonValueKind.Number)
        throw new FormatException("totalGross is not a number");

    var gross = raw["totalGross"].GetDecimal();
    if (gross < 10.00m || gross > 4999.99m)
        throw new FormatException($"totalGross outside the range: {gross}");
    if (gross.Scale != 2)
        throw new FormatException($"totalGross: two digits after the decimal point required, got {gross}");

    return new Order(number, email, gross);
}

This is not defensiveness just in case:

  • Every one of these assertions fired within my first hour.
  • The parser stops at the first error, so the JSON line below failed on currency.
  • The other three failure classes showed up on other lines of the same run.
  • alreadyApproved starts with order numbers from the prompt examples and previously approved files.
{"orderNumber": "ORD-2023-000101", "customerEmail": "anna.kowalska@gmail.com", "totalGross": 1249.9, "currency": "PLN"}

The review tool run looks like this:

$ dotnet run --project tools/OrderFixtureReview

lines read:             28
rejected:                5
approved for review:    23

line 7:
  - field outside the contract: currency
line 12:
  - orderNumber: duplicate ORD-2023-000101
line 18:
  - customerEmail outside the list: anna.kowalska@gmail.com
line 21:
  - totalGross: two digits after the decimal point required, got 1249.9

The run caught four failure classes, but not four errors in one row. The JSON above was rejected for the extra currency field before the parser reached its other values. Another line copied ORD-2023-000101 directly from the prompt example, one used the gmail.com domain even though it was not on the list and can actually receive email, and one had the wrong scale: 1249.9, with one digit after the decimal point instead of two. Separately, in another run, I got "totalGross": 1249,90 with a comma, and it failed in JsonSerializer, which is the best possible moment.

Only data that has passed through the parser and past my own eyes reaches the repository:

fixtures/
  orders.approved.jsonl

The file goes through review like any other code because it is code: it appears in the diff, its changes are visible, and I can return to it three months later. The script that calls the API sits next to it in tools/, and I run it manually at my desk. The test suite does not know the OpenAI address and is not allowed to access the network.

The seed remains where it was in September 2021, and the file from the model is only one of its inputs:

public sealed class OrderSeedFixture : IAsyncLifetime
{
    private readonly IContainer _sql = new ContainerBuilder()
        .WithImage("mcr.microsoft.com/mssql/server:2019-latest")
        .WithEnvironment("ACCEPT_EULA", "Y")
        .WithEnvironment("MSSQL_SA_PASSWORD", SaPassword)
        .WithPortBinding(1433, true)
        .WithWaitStrategy(Wait.ForUnixContainer().UntilPortIsAvailable(1433))
        .Build();

    public async Task InitializeAsync()
    {
        await _sql.StartAsync();
        await MigrationRunner.RunAsync(ConnectionString);

        // customers from a seeded factory: they define the allowed address list
        await BulkInsert.CustomersAsync(ConnectionString, new CustomerFactory().Build(3));

        // orders from the approved file, not from an API call
        await BulkInsert.OrdersAsync(ConnectionString, ApprovedOrders.Load("fixtures/orders.approved.jsonl"));
    }

    public Task DisposeAsync() => _sql.DisposeAsync().AsTask();
}

I start the container with the 3.x line of the Testcontainers library, using the new ContainerBuilder whose basics I described in MSSQL from a test. The order is the same as it was two years ago, with no room for shortcuts: migrations first, then customers from the factory, and finally orders from the file. Reversing the order guarantees a foreign key violation because an order has nothing to attach to. Once the database is running, integration tests enter through WebApplicationFactory, and I separate state between classes with xUnit fixtures exactly as I explained in my note on xUnit and NUnit. I check the list screen with Playwright 1.35.0, which also reads the same approved file, so the table assertion depends on the repository’s contents rather than on what the model invented on the day of the run.

Price and model pin

The bill is the main reason I am returning to this subject after two years. gpt-3.5-turbo costs 0.002 dollars per thousand tokens, ten times less than the line I used for the 2021 experiment. Generating two hundred candidate rows is now an expense I do not notice on my statement, and that changes the calculation: I no longer wonder whether trying is worthwhile.

It does not change the calculation for a call inside a loop. Generating data on every test suite run means paying on every run for something I will immediately discard anyway, while also bringing someone else’s uptime into my own red or green signal. Once, manually, into a file. Zero times in CI.

I pin the model the same way I pin image tags. My script uses the gpt-3.5-turbo-0301 snapshot rather than the gpt-3.5-turbo alias because the alias moves underneath me, and then I cannot tell whether next month’s result differs from today’s because of my prompt or a silent model replacement. The pin does not make responses repeatable because this API offers no such repeatability. It gives me one fewer variable when comparing two runs.

Two days ago, on 13 June, function calling entered the API together with the gpt-3.5-turbo-0613 and gpt-4-0613 snapshots: a function schema is declared in the functions field, and the model returns arguments as JSON matching that schema. I can see how this might eventually help me with data because half of my rejections are format drift rather than bad content. I am not moving anything to it today or building this post’s pipeline on it. Two days is not enough experience to pin a tool that generates input, and the parser from the previous section would have to exist anyway because schema compliance is not the same thing as compliance with a business rule.

Plan B for a situation where gpt-3.5-turbo persistently breaks one field is manual: I paste the same contract into the ChatGPT Plus interface, inspect twenty lines with my own eyes, and copy what passed. This is a desk task, not a script step and not a CI pin. For the gpt-4 API, I have a form on the waitlist and nothing more, so none of my automation can call it today.

Pitfalls that cost me an evening

The list is short because I saw every one of these things in my own input during a single evening.

Hallucinated uniqueness. I ask for thirty unique order numbers and receive twenty-seven unique ones plus three duplicates, while the model has no signal that anything is wrong. “Unique” is a condition it cannot verify because it does not see a set, only its own last token. The set is on my side, which is why alreadyApproved in the parser also holds numbers from previously approved files, not only from this one response.

A broken foreign key, supplied with complete confidence. I received several addresses outside the list, and they were all credible: valid syntax, a first and last name matching the style of the others, and a domain that was not in the prompt. Inserting such a row fails on the foreign key, and that is the good scenario. The bad scenario is when someone once “temporarily” removed the constraint in the test environment: the row goes in, the test passes, and the orphan in the child table is found a month later.

Data that looks too real. The model is very eager to produce something that looks like a real person at a real address in a real email domain. With twenty rows to review, it is visible. With two hundred, it is not, so the rule is strict and enforced by the parser rather than my vigilance: documentation domains only, everything else is rejected. The same applies to amounts and numbers that look as though they were copied from a real system.

Locales and separators. 1249,90 instead of 1249.90, pl_PL instead of pl-PL, an amount as a quoted string, or an amount with three digits after the decimal point. The first breaks parsing immediately, which is a comfortable situation. The last passes through JsonSerializer without blinking and only fails at the decimal(10,2) column or, worse, gets rounded silently.

Off-by-one errors in dates. This is why my Order has three fields and none of them is a date. When I asked for orders “from the last thirty days,” I got 31 June, one date in the future, and a midnight boundary in a time zone we had not specified. I calculate dates in code relative to an explicitly supplied point in time, just as I did in the 2021 factory. The model does not know the calendar. It knows the shape of a date.

Confidence in incorrect content. Every error above arrived without any qualification in the response. There is no low-confidence signal on which I could base an automatic gate. That is why the gate is deterministic and written by me, while the model receives exactly as much trust as a text file from an unknown author.

Summary

After a week of digging into this, I have a conclusion that fits into one sentence: gpt-3.5-turbo is an inexpensive candidate-row generator, not a source of test data.

I am breaking this down into three sentences that I want to be able to read a year from now. Chat Completions with a system role holds the output format noticeably better than the 2021 few-shot prompt, so less time goes into piecing responses together, and with a price ten times lower, trying has stopped being a budget decision. None of this changes the layer below: uniqueness, foreign keys, and business rules remain mine, checked by a deterministic parser rather than trust in the model. The schema on one side and the database reset on the other remain the sources of truth, with an approved file in the repository between them that goes through review like code.

What does not follow from this? It does not mean I have a new tool in the pipeline, because the test suite still does not call any API. It does not mean I no longer need a seeded factory: for a thousand repeatable records supporting assertions, I still use the factory because it is free, immediate, and identical on every run. Nor does it mean I have access to gpt-4 in the API, because I have a waitlist form, while the Plus interface is a desk task.

The split I am adopting by default from today is this: bulk, repeatable data from a factory; a few dozen “human-looking” rows from the model for a person to inspect; isolation from a container; and the decision about what is valid from a contract written in code. I remain the owner of the test data because I put my name to the file that enters the repository, and you can only put your name to something you have read.

Next month, I want to examine something adjacent that came up along the way: whether the same model can help me read a run’s result, not only feed it. Flaky tests and grouping similar failures are a problem where I have far more data than I have test data, because those records sit in the logs from every build. Until then, prettier data does not fix my process. It only makes an underspecified schema visible sooner.