{"id":4893,"date":"2026-08-21T12:40:16","date_gmt":"2026-08-21T12:40:16","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/08\/21\/production-grade-ai-eval-systems-what-i-learned-putting-llms-on-call\/"},"modified":"2026-08-21T12:40:16","modified_gmt":"2026-08-21T12:40:16","slug":"production-grade-ai-eval-systems-what-i-learned-putting-llms-on-call","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/08\/21\/production-grade-ai-eval-systems-what-i-learned-putting-llms-on-call\/","title":{"rendered":"Production-Grade AI Eval Systems. What I Learned Putting LLMs on Call"},"content":{"rendered":"
\"\"<\/div>\n

\"\"<\/p>\n

It was around 11 p.m. on a Thursday. Our AI support agent had been live for three weeks. Latency: Green. Error rate: Green. Then a customer DM landed in Slack with a screenshot \u2014 the bot had cheerfully invented a refund policy for a product we had never sold. Made up the SKU. Made up the rules. Returned a confident answer in 1.2 seconds.<\/p>\n

Every SRE metric said the system was healthy. The system was lying to customers at scale, and we had no signal. That night I started building what I now call a production-grade eval system. This is the version I wish someone had written for me before that Thursday.<\/p>\n

The Honest Problem<\/h3>\n

For 15 years, the SRE playbook worked because systems were deterministic \u2014 same input and output. LLMs break that contract. Your vendor can silently push a new model checkpoint on a Wednesday, and your agent develops a new personality. A re-indexed retrieval store can send \u2018what\u2019s your return policy?\u2019 to a marketing blog post instead of the actual policy doc. None of those registers as a 4xx.<\/p>\n

The result: Dashboards all green; product quietly degrading. Everything below exists to close that gap.<\/p>\n

The Three Places You Must Evaluate<\/h3>\n

Most teams vibes-check their AI feature: A PM tries six prompts, says \u201cfeels good\u201d and ships. There\u2019s no regression suite, no quality baseline and no rubric. The right fix is to evaluate in three explicit places:<\/p>\n

\"\"<\/p>\n

Most teams have Phase 1, sort of. Almost nobody has Phase 2. Phase 3 is where the screenshots-in-Slack live. If I could pick only one, I\u2019d start with Phase 3 \u2014 it catches failures you didn\u2019t predict. You need all three to call yourself production-grade.<\/p>\n

The Four-Layer Evaluator Stack<\/h3>\n

There is no single quality metric. What works is layers \u2014 cheap checks at the bottom, expensive judges at the top \u2014 with sampling upward.<\/p>\n

\"\"<\/p>\n

Layer 1 (deterministic code) runs on every trace \u2014 free. Layers 2\u20133 run on a sampled subset; LLM judges cost real money and will eat your budget if you let them run inline. Layer 4 (domain checks) runs with humans in the loop \u2014 slowest, most valuable.<\/p>\n

The Flywheel: Where Good Evals Come From<\/h3>\n

Nobody hands you a good eval set. Vendors sell generic benchmarks that score 95% on your system because they test nothing customers care about. Real evals come from your own production failures.<\/p>\n