{"id":4726,"date":"2026-08-04T10:13:58","date_gmt":"2026-08-04T10:13:58","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/08\/04\/so-agentic-systems-are-messing-up-your-slo-framework\/"},"modified":"2026-08-04T10:13:58","modified_gmt":"2026-08-04T10:13:58","slug":"so-agentic-systems-are-messing-up-your-slo-framework","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/08\/04\/so-agentic-systems-are-messing-up-your-slo-framework\/","title":{"rendered":"So Agentic Systems Are Messing Up Your SLO Framework"},"content":{"rendered":"
\"\"<\/div>\n

\"\"<\/p>\n

Engineering teams running agentic systems in production tend to discover the same gap \u2014 in the same order. First they notice that the dashboards look healthy \u2014 then realize that a healthy dashboard and a correctly behaving agent are not the same thing. It is somewhere in that apparently healthy system that an agent had been quietly producing outputs that fell outside any acceptable range of correctness for hours before anyone noticed. Not because the monitoring failed to fire, but because it was watching for the wrong thing.<\/p>\n

Service level objectives (SLOs) were built on a contract that most engineering teams have never had a reason to question. Typically, a request would come in, the service would process it and reliability was measured against whether the outcome matched the expectation within the defined thresholds. That contract worked because the systems it governed were deterministic. The same input produced the same output, and deviation from that expectation was unambiguously a failure.<\/p>\n

\u201cAgentic systems have broken that contract without replacing it with anything most platform teams are ready to operate against,\u201d says Shahid Ali Khan of TestMu AI (formerly LambdaTest). Identical inputs can produce different, plausibly correct outputs. Quiet failures complete successfully while doing something entirely unintended \u2014 and the metrics most teams are watching \u2014 latency p99, error rate and availability \u2014 tell you whether something broke but not whether the agent deviated from the expected behavior in ways that matter to the business.<\/p>\n

Khan points out that the frameworks have not caught up and the teams discovering that gap are predominantly doing so through incidents rather than through deliberate architectural decisions made before those incidents occurred.<\/p>\n

Identical Inputs No Longer Guarantee Identical Outputs<\/h3>\n

Ronak Desai, CEO and co-founder of Ciroos and formerly SVP and GM at Cisco, identifies the structural flaw in existing SLO frameworks with a precision that practitioners who have not yet run agentic systems in production may not have encountered directly. \u201cTraditional SLOs are dependent on deterministic contracts,\u201d Desai argues. \u201cA request either meets a latency threshold, or it doesn\u2019t. Agentic systems violate that assumption because identical inputs can produce different, plausibly correct outputs or quiet failures that never appear as errors.\u201d The implication is not that reliability is unmeasurable in agentic systems. It is that the measurement model has to change fundamentally rather than incrementally.<\/p>\n

The first practical consequence of that reframe is accepting that existing metrics need to be supplemented rather than replaced. Infrastructure metrics still matter and should not be abandoned. What they cannot do is tell the whole story when part of the system is probabilistic by design. That is where the observational gap opens up and where most teams are currently flying blind without realizing it.<\/p>\n

Khan, principal engineer\u2013DevOps at TestMu AI, an AI-native software testing platform, arrives at the same diagnosis from the operational side. When his team started integrating agentic components into their test orchestration platform, the existing SLO metrics proved necessary but insufficient. \u201cThese metrics tell you if something broke,\u201d Khan explains, \u201cnot why or how badly the agent deviated from expected behavior.\u201d A service can be technically up, passing every infrastructure check the team has instrumented for, while simultaneously producing outputs that fall outside any acceptable range of correctness. The gap between those two states is precisely what traditional SLO frameworks have no mechanism to surface.<\/p>\n

Closing that gap requires building a second layer of measurement on top of the first rather than trying to stretch existing metrics to cover behavior they were never designed to capture. That architectural insight, that behavioral correctness requires its own measurement layer rather than a refinement of infrastructure metrics, is the principle that connects every practitioner who has worked through this problem in production.<\/p>\n

Arun Anbumani, principal cloud infrastructure engineer at Oracle, adds a dimension to the nondeterminism problem that the agent-focused framing alone cannot capture. Hardware and firmware variability at the infrastructure layer, queue pressure on accelerators, firmware resets and resource contention on GPUs and SmartNICs introduce behavioral variance before any AI component is even involved. \u201cReliability is no longer just about whether an API call succeeds,\u201d Anbumani observes. \u201cIt is about whether the entire platform stack is behaving normally.\u201d In environments running heterogeneous infrastructure alongside agentic workloads, the nondeterminism problem compounds across layers simultaneously and platform teams that treat it as purely a model problem will keep finding infrastructure-layer failures that their agent-focused monitoring was never positioned to catch.<\/p>\n

The actionable consequence here is that SLO redesign for agentic systems has to start at the infrastructure layer rather than at the model layer. Teams that jump straight to behavioral SLOs without instrumenting the hardware and firmware signals feeding the models will build a measurement system with a blind spot at its foundation. Getting the platform-level indicators right is the prerequisite for everything that comes above it.<\/p>\n

Behavioral Correctness Needs Its Own Measurement Layer<\/h3>\n

The response that has emerged from practitioners who have worked through this problem is a layered SLO model that retains classical reliability metrics at the infrastructure layer while adding new measurement primitives for inference reliability and behavioral correctness above it. Khan and Ihor Zakutynskyi, chief technology officer at FORMA by Universe Group, arrived at this architecture independently, which is a strong indication of emerging practitioner consensus rather than individual preference.<\/p>\n

Khan\u2019s implementation adds a behavioral correctness layer above the classical reliability metrics his team retained. \u201cWe track semantic similarity scores between generated tests and a baseline corpus, confidence thresholds from underlying models, and outcome variance over rolling windows,\u201d he explains. \u201cIf variance exceeds a threshold even when the service is technically up, that is an SLO breach.\u201d The behavioral layer is not a replacement for infrastructure monitoring. It is the layer that makes infrastructure monitoring meaningful for services where the output is probabilistic rather than deterministic, and it gives teams a mechanism for detecting degradation before it surfaces as a customer-facing incident because behavioral drift tends to precede visible failure rather than coincide with it.<\/p>\n

The practical recommendation that follows is to define behavioral thresholds empirically from production data before setting them as SLO targets. Starting with rolling window variance monitoring and adjusting the threshold based on what the team observes over several weeks gives a more defensible baseline than picking a number from theory and hoping it holds in production.<\/p>\n

Zakutynskyi\u2019s three-layer model maps closely to Khan\u2019s approach and adds a precision that practitioners building these systems for the first time will find useful. The infrastructure reliability layer retains familiar metrics governed by conventional error budgets. The inference reliability layer tracks model-specific signals including token-level latency, response time variance and fallback frequency. The behavioral SLO layer enforces bounded consistency, validating that outputs meet structured contracts and measuring semantic drift against embedding-similarity baselines.<\/p>\n

\u201cA service might be 99.9% available at layer one yet still violate its behavioral consistency SLO if outputs wander beyond the defined variance envelope,\u201d Zakutynskyi notes. The reframe from uptime to what he calls \u2018trust over time\u2019, covering correctness, safety and consistency, is the architectural shift that makes SLOs meaningful for agentic systems rather than simply applicable to them.<\/p>\n

Keeping each layer tracked independently rather than aggregated is what makes incident routing possible, and collapsing all three into a single composite score obscures the causal structure of failures and makes it impossible to route incidents to the right team with the right context. Infrastructure failures route to platform engineers, while inference failures route to the ML team and behavioral correctness failures route to whoever owns the agent\u2019s goal specification and output contracts. The separation is not administrative tidiness but the operational prerequisite for a functioning response process.<\/p>\n

Start With Step Count and Build From There<\/h3>\n

Desai offers the most actionable framework for practitioners trying to operationalize agentic SLOs. It organizes around four dimensions that together form a reliability model built for nondeterministic systems rather than adapted from deterministic ones.<\/p>\n

The first dimension is latency measured by task class, specifically time per logical conclusion rather than time-to-first-byte, a metric that becomes meaningless when a task spans dozens of model calls. The second is step count as a reliability signal, where excessive steps indicate that an agent is confused or operating inefficiently. \u201cIt should not take an agent 20 steps to solve a two-step problem,\u201d Desai notes, and that observation is more immediately relatable to engineering leaders than any abstract discussion of nondeterminism because it translates directly into a metric that any team can instrument without building new infrastructure.<\/p>\n

The third dimension is probabilistic correctness, replacing binary correctness tests with pass@k-style distributional accuracy targets so that SLOs capture correctness across repeated runs rather than a single snapshot. The fourth is confidence and intent alignment, treating low-confidence actions in high-stakes contexts as first-class reliability events that automatically trigger human review or rollback rather than silently proceeding.<\/p>\n

The implementation sequence this framework implies is worth making explicit. Start with step count monitoring because it requires the least new infrastructure and surfaces the most obvious failure mode. Add task-class latency next because it gives the time-per-logical-conclusion signal without requiring behavioral analysis. Layer-in pass@k correctness evaluation once there is a corpus of validated outputs to measure against. Then, build confidence and intent alignment alerting last because it requires the most domain-specific calibration to avoid generating false positives that erode trust in the alerting system itself.<\/p>\n

Decision Points Are the Unit of Observation, Not Requests<\/h3>\n

The measurement model is only as good as the observability infrastructure feeding it, and the observability infrastructure most platform teams have is not built to surface the signals that agentic reliability requires.<\/p>\n

Khan\u2019s team has moved to probabilistic tracing as the foundation. Standard distributed tracing assumes that a request follows a predictable path. With agents, the path itself is a decision. \u201cWe emit trace annotations for decision points,\u201d Khan explains, \u201cwhich tool was invoked, what reasoning chain was followed, what confidence score triggered the choice.\u201d This gives his team the ability to correlate reliability failures with specific behavioral patterns rather than just infrastructure faults. Behavioral drift dashboards that compare current agent output distributions against baselines from previous weeks have proven more predictive of incidents than traditional alerting, triggering human review before customers notice degradation.<\/p>\n

The decision point is the unit of observation in an agentic system, not the request. Tracing infrastructure that cannot record what the agent decided and why it decided it cannot support the kind of incident attribution that agentic reliability requires. Building that tracing capability before the system goes to production rather than retrofitting it after the first incident is the single most high-leverage investment a platform team can make when onboarding agentic workloads.<\/p>\n

Zakutynskyi\u2019s observability approach logs every request with its exact prompt, model version and policy tags alongside intermediate reasoning traces and tool calls. This separation of transport-level failures from AI decision errors is what makes schema validation and trace-based replays possible, allowing his team to pinpoint whether a bad result came from network noise or from the agent\u2019s logic. Anbumani adds platform-level indicators to the observability picture, covering accelerator health, reset frequency, retry amplification and resource pressure alongside standard request metrics, making the full platform stack visible rather than just the application layer.<\/p>\n

Observability for agentic systems has to be designed into the data model from the start. Prompt logging, model version tagging and policy annotations are not add-ons that can be layered onto an existing observability stack. They require the system to emit structured data at the point of decision, which means the observability requirements have to inform the agent architecture rather than follow from it.<\/p>\n

Error Budgets in an Agentic World<\/h3>\n

The error budget question is where existing frameworks show their deepest limitations and where most platform teams are still operating without a clear answer. \u201cThe error budget model we inherited assumes failures are attributable and bounded,\u201d Khan observes. \u201cWhen an agentic component is in the causal chain of an incident alongside human-authored services, attribution becomes genuinely difficult. Did the agent fail because the model drifted, because the input it received was outside its training distribution or because a human-authored service upstream gave it bad context to reason from? Each of those failure modes has a different owner and a different fix, but the incident looks identical from the outside.\u201d At TestMu AI, the team has moved toward separate budget categories for infrastructure failures, inference failures and behavioral correctness failures, tracking each independently rather than aggregating them into a single error budget that obscures the causal structure of what is actually going wrong.<\/p>\n

Error budget reviews have to change in format as well as content. A review that aggregates all failure types into a single burn rate number cannot support the conversation that agentic reliability requires, which is a conversation about which category of failure is consuming budget, why and which team owns the response. Separating the budgets is what makes that conversation possible, and building the attribution tooling that populates each budget accurately is the prerequisite for the separation to be meaningful rather than cosmetic.<\/p>\n

The behavioral correctness budget in particular has no industry standard to calibrate against, and defining acceptable thresholds empirically from production data is currently the only option available to most teams. That is an uncomfortable position for organizations used to having external benchmarks to validate their reliability targets against, and it is the clearest signal of how early the field still is in developing the shared infrastructure that mature reliability engineering requires.<\/p>\n

What the Field Still Needs<\/h3>\n

The gap that Khan identifies from his team\u2019s experience building these systems is the same gap that every contributor to this piece identifies from their respective vantage points. \u201cWe still lack good industry standards for what acceptable nondeterminism looks like,\u201d Khan observes. \u201cWe have defined these thresholds empirically, but there is no equivalent of 99.9% availability for agent correctness. That is an open problem the industry needs to solve.\u201d The layered SLO model, the four-dimension framework, the probabilistic tracing and the behavioral drift dashboards are all real and deployable responses to a real problem. But they are responses built in the absence of the shared standards that would make them interoperable, auditable and comparable across organizations.<\/p>\n

The convergence that Khan and Zakutynskyi reached independently on the layered SLO model is the kind of empirical evidence that standards bodies and platform engineering communities should be building from. Translating that convergence into shared specifications is unglamorous work that does not generate the benchmark numbers that make AI adoption announcements compelling. But it is the work that determines whether the next generation of agentic infrastructure is built on a foundation that holds, and that work is available to any team willing to contribute what they have learned to the conversation rather than keeping it as internal knowledge.<\/p>\n

Read More<\/a><\/p>\n

\u200b<\/p>","protected":false},"excerpt":{"rendered":"

Engineering teams running agentic systems in production tend to discover the same gap \u2014 in the same order. First they […]<\/p>\n","protected":false},"author":1,"featured_media":4727,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[5],"tags":[],"class_list":["post-4726","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/4726","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/comments?post=4726"}],"version-history":[{"count":0,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/4726\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media\/4727"}],"wp:attachment":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media?parent=4726"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/categories?post=4726"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/tags?post=4726"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}