It was around 11 p.m. on a Thursday. Our AI support agent had been live for three weeks. Latency: Green. Error rate: Green. Then a customer DM landed in Slack with a screenshot — the bot had cheerfully invented a refund policy for a product we had never sold. Made up the SKU. Made up the rules. Returned a confident answer in 1.2 seconds.
Every SRE metric said the system was healthy. The system was lying to customers at scale, and we had no signal. That night I started building what I now call a production-grade eval system. This is the version I wish someone had written for me before that Thursday.
The Honest Problem
For 15 years, the SRE playbook worked because systems were deterministic — same input and output. LLMs break that contract. Your vendor can silently push a new model checkpoint on a Wednesday, and your agent develops a new personality. A re-indexed retrieval store can send ‘what’s your return policy?’ to a marketing blog post instead of the actual policy doc. None of those registers as a 4xx.
The result: Dashboards all green; product quietly degrading. Everything below exists to close that gap.
The Three Places You Must Evaluate
Most teams vibes-check their AI feature: A PM tries six prompts, says “feels good” and ships. There’s no regression suite, no quality baseline and no rubric. The right fix is to evaluate in three explicit places:
Most teams have Phase 1, sort of. Almost nobody has Phase 2. Phase 3 is where the screenshots-in-Slack live. If I could pick only one, I’d start with Phase 3 — it catches failures you didn’t predict. You need all three to call yourself production-grade.
The Four-Layer Evaluator Stack
There is no single quality metric. What works is layers — cheap checks at the bottom, expensive judges at the top — with sampling upward.
Layer 1 (deterministic code) runs on every trace — free. Layers 2–3 run on a sampled subset; LLM judges cost real money and will eat your budget if you let them run inline. Layer 4 (domain checks) runs with humans in the loop — slowest, most valuable.
The Flywheel: Where Good Evals Come From
Nobody hands you a good eval set. Vendors sell generic benchmarks that score 95% on your system because they test nothing customers care about. Real evals come from your own production failures.
- Pull production traces — bias toward anomalies, low-confidence scores and customer complaints.
- One person labels them — consistency beats coverage. Two labelers create noise that looks like signal.
- Cluster failure modes — hallucination? tone? wrong tool call? Each mode needs its own judge.
- Promote failures to regression tests — the failure can never happen silently again.
When you lack data, generate only inputs synthetically, then run your actual application on them. Never let an LLM generate both sides — you’ll get a data set that scores 99% and means nothing.
RAG: the Three Metrics That Matter
If you’re running retrieval-augmented generation — and most of you are — every interesting metric is a relationship between the question, the retrieved context and the generated answer.
- Retrieval Tier: Context precision and recall. If this is broken, nothing downstream matters. Start here.
- Core RAG Tier: Faithfulness (model stays grounded in context) and answer relevance (model answers the actual question).
- Diagnostic Tier: Citation accuracy, noise sensitivity, per-chunk hallucination rate. Run these on investigations, not alerts.
Guardrails Vs. Evaluators
This confusion causes more bad architecture than anything else I see in production AI today.
Keep them in separate code paths. Draw them in separate boxes on your architecture diagram. A guardrail that takes three seconds is a broken product. An evaluator that takes three seconds on 5% of traffic is healthy.
The Production-Readiness Checklist
If someone asks me whether their AI product is ready for production, I walk through this list:
- Golden regression data set — in the repo, version-controlled, owned by a named human.
- CI eval gate — blocks merges when quality drops on any PR touching prompts, retrievers or model config.
- Online evals on sampled production traces — at least Layers 1–2 on every span.
- LLM judges validated against humans (Cohen’s kappa >0.7), re-validated on every model upgrade.
- Weekly annotation session with a written agenda — not random sampling.
- Pinned model versions for production model and judge model. Vendor upgrades are deploys.
- Guardrails and evaluators in separate code paths.
- Someone whose job is to turn findings into roadmap items.
The Mindset Shift
Reliability for AI is not measured in uptime, it’s measured in quality of output over time. The dashboards your SRE team already built are necessary but no longer sufficient.
The new layer is the eval system — multi-tier, fed by an error analysis loop, wired into CI and live traffic, with calibrated human-in-the-loop judges. Build it, and AI stops being the system your team is scared to put on the on-call rotation.
That Thursday night was almost a year ago. We haven’t had another one like it. The dashboards are still green — and now I know what green actually means.