The moment that convinced me agents shouldn't decide

It's a few hours before a delivery deadline and the work has piled up. Somewhere in that pile is a piece that's about to miss its deadline, unless someone pushes a different piece back to make room. That's the exact moment Shot-Delivery Guardian was built for.

The system, built for Google's Agentic Cinema hackathon, watches post-production work move through editing, effects, color, quality checks, and handoff to the streaming service. When a deadline is at risk, it figures out why and recommends which piece of work is safe to push back. In real tests against the running system, it picked the same answer a fixed rule would pick 3 times out of 5. Zero broken rules across every test. It never touched a piece the client had signed off on, never touched something other work was waiting on, never touched what the director had flagged. The AI never got an important decision wrong, because it was never allowed to make one.

The final choice comes from a short, fixed list of rules written as plain code, not from the model's judgment:

def score_bump_candidates(shots):
    eligible = [
        s for s in shots
        if not s["client_approved"]
        and not s["director_flagged"]
        and not has_downstream_dependents(s, shots)
    ]
    return sorted(eligible, key=lambda s: slack_hours(s), reverse=True)

Throw out anything the client signed off on, anything the director flagged, anything other work is waiting on. Pick what's left with the most slack before its own deadline. The model runs this rule and reports the answer back, word for word. It doesn't get to argue with the filter or dress the answer up to sound more confident than it is.

The pattern: rules in code, the model in the loop

That little filter is the centerpiece, but the architecture around it is what makes the pattern portable. Eight small services, one each for intake, rendering, color, quality checks, delivery, scheduling, a test generator, and the AI itself. Queues between them, because work moves at different speeds in different stages and a slow spot shouldn't block everything behind it. Every service reports the same way, so there's no separate reporting system per tool.

Three kinds of monitoring, because one wasn't enough. One view shows how bad the backlog is overall. Another shows which specific pieces are involved and what's known about them, like who approved them and when they're due. A third shows exactly where, inside a single piece's journey, the time got lost. No single view answers the whole question.

One shared connector lets the model query all three views and write its answer onto the dashboard a person already watches. It logs in on its own, so the whole thing can start automatically the moment something looks wrong. Average time from problem to written recommendation was about 44 seconds against the live system. Fast enough that the person on call gets the answer before the deadline picture changes.

Why keep the model at all if it isn't deciding? Turning "the queue is packed and barely clearing" into something a person can act on in five seconds, that's explaining, not deciding. It also does the legwork: check the first tool, then the second, then the third, run the rule, write it up. A script could do that exact sequence. It just couldn't explain itself in plain English along the way.

The best demo wasn't a clean run. During one live test, the system hit a real problem reaching one of its three tools halfway through. It said so, plainly, in its own words. It didn't hide the gap and didn't guess. It used what it could still get from the other tools, and still gave one clear answer: this exact piece of work, with about an hour of spare time before its deadline, safe to push back.

Reading the discussion around the project, the point that stuck with me: the more capable agents get, the less interesting the question becomes of whether the model can make a decision, and the more interesting the question of whether the system needs the model to make it at all. A lot of things should stay deterministic and explicit, especially once an action affects production behavior. And the author makes the right call on the honest numbers: better to show a real 60% than a demo that only ever shows the one time it worked. The two misses, it turned out, came from the agent's own output ending before it wrote the final answer, not from the scoring function picking wrong. The filter never saw a bad candidate, because it never got handed one in time. The project is MIT-licensed on GitHub if you want the full implementation.

Why governance is the production bottleneck

The discipline matters because the failures are already happening, in boring places. In September, United Airlines' customer-service chatbot told a customer, Alison Gil, that her $200 travel credit was valid for five years from the deposit date. United then told her the credit actually expired on September 27, 2026. The airline acknowledged the bot's error and sent a replacement certificate. A customer-service bot is about the most mature agent category there is, and it still produced a confidently wrong answer with real money attached.

Gartner predicts over 40% of agentic AI projects will be cancelled before the end of 2027, with insufficient risk control as a stated reason. Scale AI's READY paper puts the problem more bluntly: an agent can perform well on benchmarks and still be unsuitable for deployment.

Key numbers

• 40%+ of agentic AI projects cancelled by 2027, per Gartner, with risk control a stated cause.
• 3 of 5 correct picks when the Shot-Delivery Guardian model chose; zero broken rules when a fixed rule decided.
• 13 of 13 test cases scored zero by a judge model whose knowledge cutoff predated the content; all passed under a faithfulness metric.
• 1 in 8 → 1 in 50 wrong-result rate for Motorway after an evaluation-first rollout.

Quick Take: the systems that survive production contact share one property. The model investigates and explains, but a deterministic gate decides whether it ships.

The counter-example, what governance looks like when it works, comes from Motorway, a UK used-car auction platform. After moving to an evaluation-first development lifecycle, wrong results dropped from one in eight agent actions to one in fifty. That's 125 errors per 1,000 actions down to 20, a 6x improvement, the kind of jump that makes an SLA credible. Motorway also lifted tool-selection accuracy from 87% to 98%, the difference between an agent that usually picks the right tool and one that almost always does.

Same models, same business. What changed was the system around the model.

The Agent DLC answer: release decisions by evaluation

The big players are voting with money on the same conclusion. OpenAI created a dedicated deployment company. Microsoft is pouring resources into Microsoft Frontier Company. AWS committed $1 billion to front-line deployment engineering. The gap between a model that works and a system that works in production is where they all think the value sits.

In June 2026, AWS published the Enterprise-Grade Agent Development and Deployment Guide, which tries to answer the question nobody in the build race was asking: how do you know you built the right thing, and how do you keep knowing it after release? Their answer is Agent DLC, the Agent Development Lifecycle.

The central mechanism is simple: release is decided by evaluation. Pass the criteria, you ship. Fail, you rework. Nobody votes at a meeting. It's CI/CD for code, except the object under test is agent behavior instead of a function.

Six phases form a closed loop. The release step is the gate. Before the gate, teams iterate between define, build, and evaluate, often dozens to hundreds of rounds before anything ships. After the gate, observation and feedback keep running: new edge cases and failure modes from production get collected and folded back into the golden standard, so the next evaluation round matches the real world more closely.

The golden standard has two parts. The evaluation specification defines what counts as correct: criteria across five dimensions, with three tiers of thresholds. The golden trajectory set is a deterministic, reproducible dataset with a baseline accuracy score, used for regression testing and continuously supplemented by production trace data. The platform can change and the model can change, but that standard is the asset a team builds once and keeps using.

The five dimensions: cognition (does the agent understand the business: intent, aliases, multi-hop relations, tool choice), quality (is output correct, complete, consistent, faithful), responsibility (can it be contained: privilege escalation, privacy, refusal boundaries, adversarial injection, audit trail), cost (where the tokens go), and performance (response speed and concurrency stability).

Each criterion sits in one of three tiers. Red-line requirements must pass 100% of the time; one veto kills the release. Gate requirements must hit a set threshold. Observation requirements just need a comparable baseline. The tiers get frozen before the first evaluation run, specifically to avoid the "look at scores, then set standards" version of grading your own homework. One caution from the paper: the five dimensions are a coordinate system for evaluation, not a component checklist for building. One dimension can involve several engineering capabilities, and one capability can affect several dimensions. Treat them as a checklist and evaluation becomes a box-ticking exercise.

Evaluation is the hardest part

Evaluation deserves the most attention because it's the one step doing two jobs at once: judging quality and deciding release. Any bias in the judge flows straight into production. Traditional software tests have deterministic right and wrong, but agent output is natural language plus multi-step behavior, and the same answer can be phrased a thousand correct ways. On top of that, the judge is usually an LLM, and it can be wrong in systematic ways.

The white paper records a real lesson: a team ran 13 test cases through a correctness evaluator and got 13 zeros. The judge model's knowledge cutoff predated the content being evaluated, so it was measuring new material with an expired ruler. Switching to a faithfulness evaluator, which checks whether the answer's claims are supported by the provided context instead of whether the judge knows the facts, all 13 passed.

Judge models come with known failure modes, all of them fixable:

Bias	What you'll see	Fix that works
Position	Swap the order of two answers and the verdict flips	Run both orders, keep only cases where both agree
Verbosity	Longer answers score higher, even padded	State in the rubric that length adds no points
Self-preference	Judge favors answers resembling its own style	Cross-validate with a judge from a different vendor
Coarse resolution	Scores pile in the middle band, nothing separates good from great	Use pairwise comparison instead of absolute scoring

There's also a hard calibration line: a judge model shouldn't be used as an automated tool until its agreement with humans reaches the level of human-human agreement. Below that, you're automating a guess.

Evaluation has real cost, and the paper is honest about it. Every trace through a judge model is a separate LLM call, and simulation adds actor fees on top. The pass standard is a rate, not a binary. A coding assistant can retry; a customer-service agent that's wrong one time in five means every fifth customer takes a real failure.

The fix priority is concrete: confirm the scores themselves are trustworthy, then check data quality, then adjust prompts and tool descriptions, and only at the end change the model. Most improvements come from prompts, tool descriptions, and retrieval strategy, not from swapping models.

Trace failures to the root, not the blame

An evaluation says "not passing." Then what? "Not passing" doesn't tell you what to fix. The paper's four-question check turns a score into a root cause. Ask in order, and the first "no" is where the bug lives.

Did retrieval get what it needed? That's context recall, and it points at the retrieval side. If yes, did the agent answer from the material? That's faithfulness, and it points at the generation side. If yes, is the material itself correct? That's correctness, and it points at the knowledge base. If yes, did it answer what was actually asked? That's response relevance, and it points back at intent understanding.

One trap here: faithfulness only checks whether the answer's claims are supported by the context. It doesn't check whether the context contained what it should have. That's recall's job, and you need both, or the diagnostic chain breaks.

Attribution then splits into three levels, each pointing at a different fix. Session-level problems, like an agent contradicting itself or losing its goal, mean fixing goal management and memory. Trajectory-level problems, like retrieving the wrong knowledge base or calling tools in the wrong order, mean fixing the retrieval pipeline and tool orchestration. Step-level problems, like a bad input type or a single wrong reasoning step, mean fixing a specific prompt or parameter. Without that layering, evaluation output stops at "low score" and never becomes engineering work.

"Escapes" are usually just sloppy security

All of this is about controlling agents on your own systems. The recent headlines about models "escaping their sandboxes" don't survive contact with reality.

Not one of the sandboxes in the news cycle was air-gapped. An air-gapped sandbox requires zero network cables, zero network interfaces, and physical isolation. What the labs actually built were soft software barriers, and the doors were left unlocked.

Take the OpenAI / Hugging Face story. The sandbox was connected to OpenAI's internal network through a package proxy. The model didn't perform magic. It found a basic flaw in the proxy and walked through an open door. The Google Gemini case is worse: testers left the model connected to the live internet during offensive tests, using a test domain that overlapped with real companies.

When I went looking for the deeper truth, the response from the community was a shrug. These are classic IT security failures: bad network segmentation, permissive egress rules, soft software barriers standing in for physical isolation. If you leave an active network interface open on a test bed, a frontier model finding its way out is sloppy cybersecurity, not emergent behavior. The escape story gets the headlines because it's scarier. The firewall-failure story is the one you can actually fix.

Governance extends beyond your own walls

Internal controls only get you so far. The moment your agent reaches someone else's platform, their rules apply, not yours.

Meta's Muse reached the top of the US App Store free chart within a week of launching. A week later, Amazon blocked it from shopping on Amazon.com.

Amazon's position: Muse didn't identify itself when browsing, appeared to capture and store customer credentials, scraped account data, and could bypass the personalization built into Amazon's shopping experience. The company told users that continued access by an unauthorized AI agent violates Amazon's Conditions of Use. Meta's position: credentials are stored securely so Muse can use them without seeing them, a separate system called Sentinel approves anything Muse sends to the internet, and the agent prompts for approval before sensitive actions like purchases.

Who's right is less interesting than what the fight signals: consent and identification are now production requirements for any agent that touches someone else's platform. Amazon's earlier fight with Perplexity's Comet browser adds a wrinkle. Amazon won a preliminary injunction in March, then lost it in August when the Ninth Circuit ruled that, under federal anti-hacking law, it was the user, not the AI company, accessing Amazon's computers. The legal terrain will keep moving.

Meanwhile OpenAI published its priorities and principles for rigorous, secure, independent third-party safety assessments of frontier models and safeguards. The noteworthy part isn't the list of principles. It's that third-party assessment is becoming a standard release gate, the way security audits became standard for SaaS.

Common pitfalls

• You let the model make the final call because it explains so well. If a decision has a defined rule, whether client approval status, contract terms, or a monetary threshold, the rule belongs in code and the model belongs in an advisory loop. That's the Shot-Delivery Guardian filter: the model reports the rule's output, it doesn't negotiate with it.
• You freeze evaluation standards after you've seen the scores. The three tiers have to be locked before the first evaluation run. Set them after and you're choosing thresholds to match whatever the agent already does, which is grading your own homework.
• You use a judge model whose knowledge cutoff predates what you're evaluating. That's exactly how 13 of 13 test cases scored zero in the white paper's case study. Check cutoff dates, or switch to a faithfulness metric that only verifies the answer against the supplied context.
• You optimize the score before checking the score itself. Trust the judge first: verify bias corrections, check judge-human agreement. Then check data quality, then prompts, then, only at the end, the model. Most gains come from the middle of that