200 OK, 1.4x the Cost: Observability for Multi-Agent Systems

The perfect run that's quietly leaking money

Your multi-agent run came back with a clean answer. Right resources, no errors. The APM dashboard shows 200 OK, latency fine, everything green.

And you were billed about 1.4x what you should have been.

That gap is the story of multi-agent observability right now. The primitives exist: nested traces, OpenTelemetry spans, token counts from every LLM call. What's rare is a data model that lets you act on them. The MAST paper ("Why Do Multi-Agent LLM Systems Fail?", arXiv:2503.13657) hand-annotated 150 traces across seven state-of-the-art multi-agent systems, hit an inter-annotator agreement of kappa 0.88, and measured failure rates from 41% to 86.7%. The uncomfortable part: most of those failures don't crash. They complete. They look fine.

Key numbers 1.4x: worst-case silent overspend per run, measured across three waste patterns 1.03x to 1.4x: the full range of that overspend 41% to 86.7%: failure rates across seven multi-agent systems (MAST) kappa 0.88: inter-annotator agreement on 150 hand-labeled traces $0: cost to run the full observability demo locally

Why APM breaks the moment an agent shows up

Traditional monitoring answers three questions: is it up, is it fast, is it erroring. For a CRUD service that's enough, because the work is deterministic and the failure modes are loud. An agent breaks all three assumptions at once.

It decides its own control flow at runtime. It calls tools in an order you didn't hardcode. And it pays per token for every reasoning step. A run can be up, fast, and error-free while doing the wrong amount of work: re-reading the same data, dragging bloated context from step to step, looping an extra cycle before it settles. None of that shows up as a 500 or a slow span. It shows up on the bill, and by then it's a trend, not an event.

AWS's own Well-Architected Agentic AI Lens frames the baseline state (its "Level 1") as exactly this problem: agent costs are visible only at the account level, Cost Explorer can't separate agents or workflows, and "teams react to billing surprises after the fact because per-agent and per-reasoning-phase attribution is missing." The target state, in AWS's words, is spending "attributable at the reasoning-cycle, agent, workflow, and tenant level rather than only at the account level."

A quick vocab anchor, since the rest of this leans on it. A span is one timed step with attributes attached: one LLM call, one tool call, one AWS read. A trace is the tree of spans for one unit of work. Classic APM records spans too, but only the loud attributes like status and latency. Agent observability puts agent-specific attributes on those same spans: cycle count, tokens, cost, and which agent owned the step. That's the whole idea. Everything else is choosing which attributes matter.

Wiring dollars into spans

Before the crew, here's the smallest version of what "wire cost into a span" means. Traccia auto-instruments LangChain, CrewAI, and the OpenAI, Anthropic, and Gemini clients, so on those stacks you get most of this for free. It doesn't yet hook AWS Strands or raw Bedrock, so you stamp the cost yourself. It's a short function:

def stamp_llm_cost(span, result, model_id="amazon.nova-pro-v1:0"):
    usage = result.metrics.accumulated_usage
    in_tok, out_tok = usage["inputTokens"], usage["outputTokens"]
    # Nova Pro, us-east-1
    cost = (in_tok / 1000 * 0.0008) + (out_tok / 1000 * 0.0032)
    span.set_attribute("llm.model", model_id)  # REQUIRED. wrong key = silently zero
    span.set_attribute("span.type", "LLM")
    span.set_attribute("llm.usage.prompt_tokens", in_tok)
    span.set_attribute("llm.usage.completion_tokens", out_tok)
    span.set_attribute("llm.cost.usd", round(cost, 6))

That's the whole move: read the token usage the SDK already returns, convert it with real pricing, attach it to the span. The output side matters more than it looks. Output tokens on Nova Pro cost 4x input, so the agents that hurt your budget are the ones that generate: long summaries, extra verification loops, verbose tool output.

One attribute name will bite you. If the span key is anything other than llm.model, the cost silently reads zero downstream. There's no error, just an empty column.

Quick Take: A correct-looking answer is not evidence of a healthy run. The evidence lives in the trace, on attributes you put there on purpose.

Modeling a crew as a trace

The crew runs on Nova Pro through AWS Strands Agents using an agents-as-tools pattern. A supervisor named investigation_run delegates to three specialists. Each specialist owns several read-only tools, and every tool opens its own live span, so the dashboard shows the agent, then each AWS read nested under it with a real duration.

The delegation is intent-routed, not fan-out-everything. The supervisor's instructions are strict: call only the specialist whose domain the user actually asked about. Ask only about cost and only the Cost Analyst runs, while Health & Ops and Security Auditor never spend a token. Ask a whole-account question and all three light up. You can watch this in the trace: agent.delegated_to records which specialists ran, and a shared session.id ties the four agents into one investigation.

Agent	Role	Tools	AWS APIs
investigation_run	supervisor, decides scope and synthesizes	delegates via agents-as-tools	none directly
cost_analyst	read-only FinOps	cost_forecast, last_month_cost, daily_cost_trend	ce:GetCostAndUsage, ce:GetCostForecast
health_ops	read-only SRE	account inventory and health signals (5 tools)	EC2, CloudWatch, S3, Lambda reads
security_ops	read-only security review	GuardDuty and IAM posture reads	guardduty, iam (read-only)

The Cost Analyst is the useful example. On a real run it reports actual month-to-date spend, the forecasted month-end total, top services by spend, the month-over-month direction, and the single most expensive day compared against the daily average. Three tool spans, each with its own duration and its own dollar cost. That per-agent cost figure is exactly what turns a "successful" run into a caught overspend later. The full wiring lives in src/crew.py in the companion repo if you want to steal it.

Catching silent waste

This is the payoff section. Three waste patterns, each reproducible with real dollars:

1. Redundant re-reads. The agent fetches the same S3 listing or CloudWatch metric twice because it doesn't check what it already has. Every re-read is green on the dashboard and costs real tokens.
2. Context bloat. Earlier steps drag their full context forward, and input tokens stack per agent per step. Input is the cheap direction on Nova Pro, but "cheap" is relative when it compounds across a whole crew.
3. Extra loop cycles. The agent settles on an answer, then runs one more verification cycle to be safe. Each cycle is a full LLM call, mostly output tokens, the expensive direction.

Measured across the three patterns: 1.03x to 1.4x overspend per run. Two runs can return the same correct answer, both show 200 OK, and one costs 43% more. Without cost attributes on spans, they're indistinguishable.

When I tested this against a real account, the surprises were mostly about the SDK, not the agents. I spent days probing Bedrock's response shape before I trusted it, watched my trace design break twice, and ended up reading the SDK source when the docs ran out. The silent-waste patterns reproduced within a few runs, with real dollars. A cost-only prompt lit up one agent and left the other two idle, and the bill difference was immediate. That's the thing nobody shows you in a quickstart: the demo looks identical to the healthy run until you attach dollars to spans.

The fix is a baseline and a delta. Record a known-good run once: expected cycle count, expected tokens, expected cost per agent. Compare every run against it. A prompt tweak that silently doubles token usage becomes visible the same day, before finance finds it on a monthly bill.

The observability stack is inverting

PostHog is the clearest sign that the category is moving. It's an open-source product analytics platform, and its newest additions are two-sided. First, native AI observability: traces, generations, latency, and cost for LLM-powered apps, sitting next to session replays, error tracking, and feature flags. Second, self-driving mode: an agent reads signals in your product data (errors, rage clicks, failed queries) and turns them into researched reports and pull requests you review and merge. The observability platform stopped being a dashboard and started being a teammate.

There's a symmetry here you don't get from classic APM. The multi-agent system needs instrumentation that speaks its language: per-agent cost, cycle counts, tool attributes. And the observability tool is becoming an agent itself: it reads telemetry, decides what's wrong, proposes a fix. PostHog even exposes an MCP server, so Claude Code or Cursor can query your analytics platform directly. Spaces like convaiinnovations/laya-demo are already hosting multi-agent apps on "Zero" infrastructure, which is the same trend from the deployment side: agents as the product, runtime underneath.

Layer	What it tracks	Failure model	What you still build
Classic APM (CloudWatch, Datadog)	uptime, latency, error rate	loud failures: 500s, timeouts	per-agent attribution
LLM tracing (Traccia, Langfuse, Arize Phoenix)	spans, tokens, cost, agent identity	silent waste: loops, bloat, re-reads	baselines and delta detection
Agentic observability (PostHog self-driving)	product signals plus LLM traces	drift between intent and behavior	review and merge discipline

The AWS-native path has the same wall I kept hitting. AgentCore Observability exports traces to CloudWatch and is the natural fit if your agents run on its runtime, but cost reporting stops at the account level, so you still assemble per-agent dollars yourself. That's why the span-level approach matters: it doesn't depend on which runtime your agents use.

Rehearse before you ship

Instrumentation tells you what happened. But observability for agentic systems has a second half: rehearsal. You need a controlled environment to establish the known-good baseline that production monitoring compares against.

MiroFish, a multi-agent prediction engine trending on GitHub, shows what that tooling looks like. It extracts seed information from real-world material (news, policy drafts, financial signals), builds a high-fidelity parallel world of thousands of agents with independent personalities, long-term memory, and behavioral logic, then lets them interact and evolve. You inject variables from a god's-eye view and watch the simulation play out. It's built on OASIS (Open Agent Social Interaction Simulations) from the CAMEL-AI team, with Zep Cloud handling memory.

A simulation platform belongs in an observability article because the failure modes that matter for agents (silent waste, looping, context bloat) live in behavioral traces, and you can only study behavioral traces at scale in a sandbox. The prediction product is the rehearsal environment; per-span cost tracking is the production instrumentation. You want both. Teams that skip the rehearsal end up doing their first experiment in production, and that's where the 1.4x bills come from.

What trips people up

Five concrete mistakes, all of which I either made or watched happen:

1. Getting the span key wrong. llm.model must be exactly that string. One character off and the cost attribute reads zero with no error. My first traces ran a full day before I noticed the cost column was empty.

2. Treating console access as IAM access. For Bedrock, enabling amazon.nova-pro-v1:0 in the model access console is a separate step from allowing bedrock:InvokeModel in your policy. You need both. Missing either one produces a confusing runtime 403.

3. Planning to attribute cost from Cost Explorer. AWS's own framework says this doesn't work. Cost Explorer stops at the account level; it can't separate supervisor overhead from worker execution, let alone one agent from another. Attribution has to happen at trace time, in your spans.

4. Treating cost as an event. The monthly bill is a trend, not an event. A single 1.4x run is a rounding error. Thousands of them is a budget line. You need baseline-delta detection per run, not a monthly reconciliation.

5. Assuming plain OpenTelemetry is enough. OTel gives you spans, not dollars. Token-to-cost conversion, agent identity, and session grouping are all vocabulary you build yourself if your tracer doesn't ship it. That's the entire reason tools like Traccia and Langfuse exist. Budget for one or budget for the hand-rolled layer.

One thing to remember

When everything is green, the trace is the only place the truth lives. A correct answer is not evidence of a healthy run. Wire per-agent identity, token counts, and dollar cost onto spans on purpose, keep a known-good baseline, and compare every run against it. That single habit catches more agent regressions than any error-rate alert ever will.

The Bottom Line

If you're putting multi-agent crews into production, adopt per-agent cost spans now. At thousands of runs per day, a 1.03x to 1.4x silent overspend is real money leaking with no alarm attached, and it's the only waste pattern you can actually fix once you see it.

If you can't change tracers this quarter, at least export token usage per agent step. The llm.usage and llm.cost attributes ride on any OpenTelemetry backend you already own, and tomorrow's baseline-delta tooling will have data to chew on.

One thing to watch: self-driving observability. PostHog's agent mode and MCP connectors are already moving dashboards toward fixes. Expect observability tools to start proposing (and merging) remediation PRs within about six months. The same way agents need observability, observability is starting to need agents.