Build a Prompt Evaluation Workflow Before You Ship

Start with behavior, not prompt wording

A production prompt is part of a feature, not a document to admire in isolation. Write down what a user is trying to achieve, the information the system can rely on, the output shape a downstream system expects, and the conditions under which the feature should stop. Those statements become the first evaluation contract.

For example, an invoice extraction assistant may need to return normalized fields, mark unreadable values as uncertain, and never invent a tax identifier. The prompt can change later. Those behaviors should not. A good evaluation workflow protects the user outcome while leaving room to improve the implementation.

• Describe the user job in one sentence.
• State the expected output schema.
• List unacceptable outcomes before creating examples.

Build a small test set with useful slices

Begin with 20 to 40 cases that represent normal requests, incomplete requests, conflicting instructions, unsafe requests, and awkward formatting. A smaller set that you understand is more useful than hundreds of synthetic examples with no reason for inclusion.

Tag every case with a slice such as short input, long input, non-English text, sensitive action, tool use, or strict JSON. When a change breaks only one slice, the tag tells you where to look. It also prevents an attractive overall score from hiding a severe failure mode.

Use rubrics that a reviewer can apply consistently

A score is meaningful only when a reviewer can explain it. For each case, define a few observable criteria: factual support, instruction following, format validity, safe refusal, and tone where tone matters. Use pass, partial, and fail rather than a vague one-to-five scale when you need a release decision.

Automated checks are excellent for schemas, required fields, forbidden phrases, and tool-call shape. They are weaker for nuanced correctness. Combine deterministic assertions with a lightweight human review process for the examples that influence product decisions.

Treat changes as experiments

Version prompts, model settings, retrieval configuration, and tool schemas together with the evaluation result. If an answer becomes worse after a change, the team needs to know what actually changed. A screenshot of a good answer is not a regression strategy.

Run the same cases before and after a proposed update. Review failures by slice, inspect a few unchanged cases, and record why you accepted a tradeoff. This is how a prompt workflow becomes engineering work instead of a cycle of intuition and surprise.

Make release decisions narrow and reversible

Do not wait for a universal quality score. Define the few failures that block a release: invalid structured output, unsafe action, fabricated evidence, or an unacceptable failure rate on a critical user path. Everything else can become a tracked improvement with a clear owner.

The first version should expose enough telemetry to learn. Log prompt version, model version, latency, token usage, tool outcomes, and a privacy-safe trace identifier. That record lets you connect an evaluation result to the system that produced it.

Checklist

• The feature contract names expected and unacceptable behavior.
• Each test case has a purpose and a slice tag.
• Rubrics separate format checks from qualitative review.
• Prompt, model, retrieval, and tool versions are recorded.
• Critical failures have a release-blocking rule.