Build the LLM Evaluation Set Before You Write a Prompt

Someone changes one sentence in a system prompt, pastes in three inputs they have used before, reads the outputs, decides they look better, and merges. Two weeks later a customer reports that the assistant now confidently answers a category of question it used to decline. Nobody can say which change caused it, because nobody measured anything.

That loop is the normal state of AI feature development, and it is why so many of these features get quietly switched off six months after launch. The fix is unglamorous. Before you write the first prompt, build an LLM evaluation set: a fixed, versioned collection of real inputs with expected outputs or a grading rubric, that you can run on demand and get a number back from.

Prompt engineering without measurement is just opinion

A prompt is code. It has behaviour, regressions, and edge cases. The difference is that when you break a function, the type checker or the test suite tells you. When you break a prompt, nothing does, because the output is still fluent English and still looks plausible.

Three manual spot checks cannot detect a regression that affects, say, one answer in twenty. They cannot detect that your new phrasing made the model more willing to discuss competitor pricing, or that adding "be concise" quietly stripped the citations. Eyeballing catches catastrophic failures and misses everything else, which is backwards, since the catastrophic ones were already going to get reported.

There is a social cost too: without a shared number, prompt decisions go to whoever argues most confidently in the review thread.

What an evaluation set actually contains

Somewhere between 50 and 200 cases is the useful range. Below 50, single cases swing the score too much to trust. Above 200, each run costs real money and real minutes, which discourages running it, which defeats the point. Start at 50 and grow.

Each case is an input plus either an expected output or the criteria a good output must satisfy. The inputs must come from your own domain. A generic benchmark tells you how a model performs on other people's problems.

Cover three categories deliberately:

• The common path. The queries that make up most of your traffic. Boring, high volume, and the ones a regression will hurt most.
• The awkward edges. Ambiguous phrasing, missing context, two questions in one message, a document that is mostly a table, a customer writing in a second language. Behaviour drifts here first.
• Inputs that should be refused or deflected. Requests for legal or medical advice, prompt injection attempts, questions the system has no data for. An expected output of "I do not have that information, here is how to reach a human" is a real test case, and the one people forget to write.

That last group keeps a feature safe when the model changes underneath you. Refusal behaviour is not stable across model versions, and if you never encoded it, you will never know when it moves.

Mine what you already have

Teams skip this step because building an eval set sounds like a project. It is not, if you stop inventing cases and start harvesting them.

Support tickets are the best source available: real questions in real customer language, already paired with the answer a human gave, already sitting in a database. Export the last few hundred, deduplicate the near identical ones, and you have your common path. For document processing, pull actual documents out of the pipeline, not clean samples. For search, use the real query log, including the malformed queries.

Then get one domain expert to grade 50 outputs, once, in a spreadsheet. Not the engineers. The person who knows what a correct answer looks like in this business. A couple of hours of their time gives you a graded baseline, and it surfaces disagreements about what "correct" means before they get baked into a prompt.

The eval set is not a test artifact. It is a written specification of what the feature is supposed to do, in a form a machine can check.

Three grading methods, and when each is right

Exact match, including close variants like normalised string comparison and JSON schema validation, applies whenever the output is structured: classification labels, extracted field values, routing decisions, tool call arguments. This grading is free, deterministic, and instant, so design your output format to fit it wherever you can. Asking the model for a category enum plus a free text explanation lets you grade the enum exactly and the prose separately.

Rubric grading with a human applies where quality is subjective and the stakes justify the cost. Write three to five criteria, each scored one to five: is it factually supported by the retrieved context, does it answer the question actually asked, is the tone right, does it cite sources. A human runs this against maybe 30 cases at a milestone, not on every commit.

LLM as judge sits in between and is the one most often misused. It works when the judging task is narrower and more mechanical than the original task. "Does this answer contain a claim not present in the provided source text" is a good judge prompt. "Is this a good answer" is not, because you have replaced an unmeasured system with an unmeasured measurement. Calibrate the judge against your human graded 50 first. If it agrees with your expert on 45 out of 50, it is useful. If it agrees on 32, you are generating noise with a decimal point on it.

Record the score per category rather than as one blended number. A single overall percentage tells you nothing you can act on, while separate scores for the common path, the awkward edges, and the refusals tell you which prompt section to open.

Retrieval failures wearing a model failure costume

In any system that pulls context before generating, most bad answers are not the model being stupid. They are the model faithfully answering from the wrong three chunks of text. Someone asks about the 2025 refund policy, the retriever returns the 2023 version because it scores higher on lexical similarity, and the model produces a fluent, well structured, wrong answer.

You cannot tell those apart by reading outputs, and the two failures have different fixes. Model failures get fixed with prompts and models. Retrieval failures get fixed with chunking strategy, embedding choice, metadata filters, hybrid search, and reranking. Teams spend weeks tuning prompts against a problem that was in the index all along.

Instrument the eval set to separate them. For each case, record the retrieved chunk IDs alongside the generated answer, and score retrieval independently: did the chunk containing the correct answer appear in the top k. When recall is perfect and the answer is still wrong, that is a generation problem. When the right chunk never made the cut, no prompt will save you. This is instrumentation worth building into the AI integration from day one, not retrofitting after complaints start.

Migration is where the set pays for itself

Model releases arrive on a schedule nobody controls. A new one claims to be faster, cheaper, and better. Should you switch.

Without an eval set, that becomes a multi-week discussion involving vibes, a few side by side comparisons, and a nervous partial rollout. With one, it is an afternoon. Point the runner at the new model, run 150 cases, compare scores per category. Suppose quality is flat, cost drops sharply, and refusals degrade on four cases: you add two lines to the system prompt and re run. That is a decision with evidence behind it.

The same machinery covers swapping the embedding model, changing chunk size, adding a reranker, or moving a step to a smaller model to cut latency. Each is a scary change to an uninstrumented system and a routine one to an instrumented system. Treating the eval runner as part of the application, checked into the same repository and runnable in CI, is a plain full stack development concern rather than a research activity.

The set grows from production

The first version will be incomplete. That is fine, because the mechanism that completes it is already running: your users.

Make it a rule that every reported failure becomes a case before the fix is written: add the input and the correct output to the set, confirm the case fails, then fix it. That is regression testing, and it means the same bug cannot ship twice. Over time the set encodes every mistake the system has ever made, which is an asset no competitor can copy.

Run it in CI on any change to a prompt, a model version, a retrieval parameter, or the chunking pipeline. Set a floor: the build fails if the score drops below the last release. That single gate converts prompt editing from something risky and unreviewable into something that behaves like the rest of your codebase.

Where to start on Monday

Pick the narrowest slice of the feature that has real users or real documents behind it. Export 50 real inputs, have one expert write what a correct answer looks like for each, and write a script that runs all 50 and prints a score. That is roughly a day of work, and everything after it is easier: prompt changes get measured, model upgrades get decided instead of debated, and retrieval problems stop masquerading as model problems. The teams whose AI features are still running two years later tend to be the ones that did this before they wrote the first prompt.