Building Reliable Prompts

Prompts held to production software standards: automatic optimization against curated datasets, guardrails for edge cases, and evals with runtime observability.

Prompts that power production AI systems should be held to the same standards of reliability, observability and maintainability as any other software component.

We learned this quickly after deploying our first prompts to production at Brevian. First, prompts are fickle and tightly coupled to the underlying model, and a prompt tuned for one model can degrade on the next. Second, prompts need to be designed for what good looks like: language models are good at generating content even at the expense of facts, and few-shot prompting with examples can backfire because the model imitates the pattern of the examples to the point of comical inaccuracy. Third, structured JSON output has come a long way, but the constraints your APIs have downstream still need to be reflected in the prompt explicitly.

To manage this we follow a structured 3-Pass Rule when designing and deploying prompts. The process combines automatic optimization, guardrails, and continuous evaluation.

Pass 1: Automatic Prompt Optimization (APO)

We begin with APO. A curated test dataset is created that contains a representative set of inputs along with their expected outputs and required formatting rules. The prompt is iteratively tuned against this dataset, using reinforcement learning, until it meets predefined success criteria across all test cases. The result is a prompt validated against real product use cases rather than ad hoc examples.

Pass 2: Guardrails and constraints

After the initial optimization, guardrail instructions are added. These explicitly define constraints and expected behaviors, such as:

• Maintaining output within a specified scope and tone.
• Handling incomplete or ambiguous inputs by prompting for clarification.
• Avoiding irrelevant, unsafe, or non-compliant responses.

This pass is what makes the prompt behave predictably on unexpected and edge-case inputs.

Pass 3: Evals and runtime observability

Finally, prompts are placed under evals: automated evaluation suites that verify outputs both during CI/CD and in the runtime environment. The evals compare live outputs against expected patterns, surface exceptions when hallucinations or format deviations are detected, and include dedicated edge-case inputs to identify regressions and model drift over time. Drift and unexpected behavior are detected before they reach end users.

The mapping to traditional engineering

For engineers accustomed to traditional software systems, the 3-Pass Rule maps to established disciplines. In classical system engineering we want a system that is observable, testable and resilient: one that handles expected workloads correctly, fails gracefully under unexpected conditions, and provides clear signals when something is wrong. In practice we build that with strong validation test suites, well-defined interfaces, and thorough monitoring. The same principles apply to prompts.

Applying these principles addresses the failure modes we see in production. Continuous evaluation highlights subtle shifts in model behavior. Outputs are checked against expected patterns, with deviations flagged. Guardrails and evals catch failure modes before they affect end users, and dedicated test inputs keep behavior consistent under stress conditions.

Prompts evolve alongside models, datasets and user needs. The discipline that keeps other production systems reliable, validation against a dataset, constraints for the edge cases and observation in production, is what keeps prompts reliable as well.

References

Teki, S. Prompting is the New Programming Paradigm in the Age of AI. Personal Blog, 2023. Link

Prompt Engineering Guide. Few-Shot Prompting. PromptingGuide.ai, 2023. Link

QED42. Building Simple & Effective Prompt-Based Guardrails. QED42 Insights, 2024. Link

Humanloop. Structured Outputs: Everything You Should Know. Humanloop Blog, 2023. Link

Wolfe, C. Automatic Prompt Optimization with Reinforcement Learning. Substack, 2023. Link