Two truths about production agents

There are two separate stories about AI agents in production right now, and they seem to contradict each other.

Story one: most "agents" are overengineered. An LLM call with a vector store bolted on, when a regex or a SQL WHERE clause would do the job in microseconds, deterministically, for free. The result works in the demo. It is also slower, more expensive, harder to debug, and nondeterministic in places where it didn't need to be.

Story two: the agents that earn the name are rewriting data center economics, and CPU is the surprise winner. Meta's Muse, a subscription that gives every user a dedicated VM and an assistant that keeps working after you close the app, topped US mobile download charts and sent CPU stocks up. Intel is hosting 1,000+ agents on a single Xeon 6+. The 2030 server CPU market forecast was revised from $59B to $210B in six months.

Both stories are true. The hard part is telling them apart, and most teams currently can't.

Intel's projection gives a sense of scale: 80% of enterprise applications will embed at least one agent by the end of 2026, and active agents in Chinese enterprise scenarios reach 350 million by 2031. At that volume, the difference between "an agent that should be a function" and "a function masquerading as an agent" stops being a style argument. It's a capacity argument.

The if-statement audit

Start with the critique, because it applies to more production code than anyone wants to admit. The dev.to piece that kicked off this thread walks through three failure modes. I've now personally run into each of them in production codebases, usually after someone else's team shipped them.

Fixed-format extraction is the classic. Invoice numbers match INV- plus eight digits. A team sends every email to an LLM asking it to extract the number. It works 98% of the time. The other 2%, the model "helpfully" reformats the number or grabs a purchase-order ID instead. Each call costs money and adds a few hundred milliseconds. A compiled regex runs in microseconds, is unit-testable, and never daydreams. The right split is regex first, LLM only on the residue: whatever the pattern didn't match.

The same category error reappears as a filter query pushed through a vector store. "Show me all unpaid orders from customer 4417 in the last 30 days" is not a semantic question. It's a filter. When I picked apart a team's knowledge bot recently, exactly this query went through embeddings, top-k retrieval, and an LLM summary. The team was surprised the answer was incomplete. It was never going to be complete: a vector hit list is a probabilistic shortlist, not an evidence path.

What you're actually asking	Right tool	What the LLM does instead
Fixed format, structured input (INV-12345678)	Regex / schema validation	Reformatting, hallucinated digits, latency, token spend
Closed-world facts (customer, date range, status)	SQL with an index	Top-k misses rows silently; nobody audits the summary
Enumerable routing logic (four branches)	If-statement / decision tree	Probabilistic routing, occasional loops, unreproducible assignments
Fuzzy meaning: paraphrase, similar complaints, messy prose	LLM / embeddings	The one place the model earns its keep

The community reaction to the original post was telling. Engineers recognized the patterns instantly. One comment I keep thinking about: if your agent has one tool and the planner calls it every time, it's a function call with a token bill. I now apply Jack Diederich's "Stop Writing Classes" rule to agents the way I do to Python: if a class has two methods and one is __init__, it's a function. If your agent has one tool and no real branching, it's a function. And I've watched teams spend ten hours wiring an agent to write a script that would have taken ten minutes by hand, then another week in a review loop on the ruleset it kept violating.

When you actually need an agent

None of this means the agent category is fake. Open-ended work, operating a GUI, navigating software with no API, planning across dozens of steps, needs an agentic model, not a bigger prompt. That's what Holo4, released this week by H Company, is for.

Holo4 refuses to specialize to one interface. Most agentic models are GUI-only or tool-call-only. Holo4 clicks and types on a screen, writes and runs code, and calls MCP and API tools, whichever fits the task, sometimes all three in a single workflow. The 27B dense version scores 61.7% on OSWorld 2.0, the hardest desktop-control benchmark. Opus 5.5 gets 81.8%, but through an API at a much higher price per task. The 35B-A3B MoE variant activates only about 3B parameters per token, which puts its inference cost near a small model's.

Model	Size	OSWorld 2.0	Practical read
Holo4 27B	27B dense	61.7%	Self-hostable on a single high-end GPU
Holo4 35B-A3B	35B total, ~3B active	30.9%	MoE pricing, but GUI control still trails the dense version
Opus 5.5	Closed frontier model	81.8%	Reference point for maximum accuracy

The efficiency signals matter more than the scores. On a Pac-Man-building task in Godot, Holo4 took 68 calls and 2.4M tokens. Its base model, Qwen3.8 27B, took 197 calls and 11.4M tokens on the same prompt. That's a roughly 5x token-efficiency gap on identical work, and it's why "just use a bigger model" isn't a substitute for training the agent loop itself.

On the open-source side, Agent Zero takes a different bet: give the agent a real Linux desktop, a real browser with DOM annotation, live document cowork, and transparent internals. Prompts live in prompts/, tools live in tools/, and plugins are inspectable and editable. It's the philosophical opposite of a black-box agent framework, and a useful counterweight to the framework-first picking that's flooding production.

Quick Take: The agents that shouldn't exist are eating GPU budgets on if-statements; the agents that should exist are why CPU just became the hottest seat in the data center.

The agent loop is a CPU loop

The hardware side is the part most ML engineers haven't internalized yet. Break down what an agent does over a task: it perceives, plans, decomposes into sub-tasks, spawns sub-agents, calls tools, verifies results, reflects, retries. The GPU handles one slice of that loop, the model inference. Everything else, orchestration, sandbox management, tool execution, memory, the reflection loop, runs on CPU. Analysts at Haitun Research put it bluntly: an agent is a CPU workload that occasionally asks a GPU for a token.

This is why Meta's Muse matters beyond the product. Each Muse user gets a dedicated Linux VM with 2 vCPUs that keeps running after they close the app. A chatbot is a transaction: prompt in, answer out, GPU idles. An agent is a tenant: it occupies a VM, holds state, and keeps executing on your behalf. The price-watching example from the analysis: "monitor Mac Mini prices" becomes 30 days of a CPU-side script checking a price API, with GPU inference only on the day the price actually drops. That inverts the demand model. The longer the task horizon, the more CPU hours per GPU call.

Intel's numbers from Connection 2026 back this up. A single Xeon 6+, up to 288 efficiency cores and 576MB of last-level cache, hosts over 1,000 agents, or creates 350 sandboxes concurrently at 200ms startup latency. Intel says "10,000 sandboxes in 10 minutes" is now a production requirement from real customers, and 80% of instructions hit in L2 at that density. The interesting architectural bet is putting compression (QAT), data movement (DSA), and matrix math (AMX) inside the CPU instead of bolting on external accelerators. Intel's argument: if acceleration is external, the round-trip between CPU and accelerator eats the benefit. For bursty agent execution, a shared multi-core L2/L3 makes more sense than an exotic memory hierarchy.

The memory wall: KV cache and the 2 GB problem

The bottleneck is memory, not cores.

An agent's working memory is a scratch pad. Every intermediate result in a multi-step task gets written to the KV cache, and contexts keep growing. Intel cited a concrete case: a 235B-parameter model with a 1M-token context produces around 300GB of KV cache. That exceeds the HBM of any GPU currently shipping. You can't fit it, and you can't afford to keep it on-device. The practical answer is tiered offload: HBM to system DDR to SSD, with compression along the way. Intel's KV Shrink toolkit reports 1.42x lossless compression and 50%+ on lossy settings, a ~9.7x transfer speedup, and up to 15x better time-to-first-token by overlapping decompression with the next layer's compute.

The numbers that matter when agents go to production:

• A 235B model at 1M-token context produces ~300GB of KV cache, more than any GPU HBM holds. Offload and compression stop being optimizations and become default infrastructure.
• A Kimi research paper measured KV cache hit rates in agent workflows at 50-70%. The misses alone can exceed a GPU's entire HBM.
• One Xeon 6+ hosts 1,000+ agents, or 350 concurrent sandboxes at 200ms startup. The binding constraint is memory, about 2GB per agent.
• A leading Chinese LLM vendor's CPU demand grew 5x year-over-year as agent workloads took off.

The software side of agent memory is maturing in parallel. Cognee, trending on GitHub this week, gives agents durable memory as a knowledge graph. The twist: you can run the whole extraction pipeline on free local small models, no API key. Text ingestion, retrieval, and session storage all work on CPU. Same philosophical thread as the Intel story: not every step of an agent needs a frontier model, and not every byte needs to live in HBM. On the BEAM conversational-memory evaluation, Cognee scores 0.79 at 100K token context and 0.67 at 10M. At the scale of a long-running agent project, recall stays usable but measurably degrades, which is exactly why memory design deserves its own budget line.

One more number for perspective on the scale shift. NVIDIA's Vera CPU rack packs 256 CPUs at 88 cores each, 22,528 cores total, which NVIDIA says supports over 22,500 sandboxes. That's more than 5x the CPU cores of a full VR NVL72 GPU rack. Read that again: the company selling flagship GPUs is also selling standalone CPU racks for agents.

Three CPU markets, one repricing

The analyst framing that clicked for me splits data center CPUs into three categories with unrelated demand drivers.

CPU class	Job	Where it lives	Demand driver
Traditional	Web servers, databases, app logic	Standard racks	Enterprise refresh; grows slowly
AI head-node	Feed GPUs, manage KV cache, handle non-accelerated code	Inside GPU servers	GPU shipments; CPU:GPU ratio climbing from 1:8 toward 1:1
Agentic	Orchestration, tool calls, sandboxes, reflection loops	Standalone CPU racks	Concurrent agent count, not GPU count

The head-node story is pure economics. With GPU utilization on the line, buying more CPU to keep the GPU fed pays for itself; ratios moved from 4:1 to 2:1 as accelerators got more complex. The agentic story is pure increment: no GPU would have been bought for that work at all. The overall ratio is now heading to 1:1, and some complex agent scenarios need denser CPU than GPU.

The market repricing reflects this. The 2030 server CPU market forecast went from $59B in February to $210B by August, a 3.6x revision in six months. ARM estimates 30M CPU cores per gigawatt of traditional AI data center capacity, and 120M per gigawatt once agentic workloads mature. Haitun's model at 100M Muse-scale users lands at 48.5M cores in the neutral case: pure GPU racks need 17.3M, the user-side fixed layer adds 1.25M, and the sub-agent elastic layer adds 30M.

The winner picture is less obvious than it looks. Intel and AMD split about 93% of today's server CPU revenue, 58/35. But agentic demand is net-new and GPU vendors want it. Haitun projects NVIDIA, ARM, and Qualcomm taking share: $30B, $10B, and $4B in incremental annual revenue by 2030 respectively. Intel and AMD see larger absolute increments, $55B and $62B, which is why their 2030 earnings expectations barely moved, around 10% upside each. This complicates the "CPU comeback" narrative for the incumbents specifically. The market triples, but their share shrinks in relative terms even as absolute revenue grows.

Common Pitfalls

Vector search where SQL belongs. If your question is exact ("did we pay invoice 4412?"), embeddings are the wrong index. A top-k list is a probabilistic shortlist with no completeness guarantee. Classify each production question as exact, filter, or semantic before choosing the path. If it's exact or filter and it still crosses embeddings plus an agent loop, you're paying GPU dollars for nondeterminism you didn't need.

LLM extraction with no regex first and no fall-through logging. The 2% of invoices the model mangles are silent data corruption. Run the regex first, send only the residue to the model, and log everything the regex misses. After a month, you'll likely find the "messy" 2% is mostly a second pattern you never wrote.

Framework-first thinking. Picking the wrong framework is fixable. Making the framework the default unit from day one, so every caller depends on it, is the expensive mistake. Keep the entry point a plain function. Grow an agent behind it only when branching and tool use actually show up.

Forgetting that memory is the constraint. At 2GB per agent, a 128-core node with a 1:2 core-to-memory ratio and 4x over-subscription needs roughly 1TB of RAM. Cores stop being the bottleneck long before memory does. Budget memory per agent before you design the fleet, and treat KV cache offload like tiered storage: it needs a plan, not a prayer.

Benchmarking the model instead of the loop. Holo4's Pac-Man case is the lesson: same task, same prompt, 2.4M tokens vs 11.4M tokens depending on training. Token efficiency is part of model quality. Measure cost per completed task, not just accuracy on a static benchmark.

One thing to remember

Around all the hardware numbers, one habit matters more than anything: look at what the task actually is before reaching for the most impressive tool in the room. If the logic can be drawn on a whiteboard, write the branches. If the agent is real, plan for what it really is. A CPU workload with a memory problem that occasionally rents time on a GPU.

The Bottom Line

If you're building a feature that extracts fixed formats, filters known facts, or routes on enumerable logic, you should use regex, SQL, or an if-statement first. Route to the LLM only on the residue, and log that residue. It's a labeled dataset waiting to be mined.

If you're running real multi-step agents, you should plan the CPU tier like a first-class citizen: sandboxes, about 2GB of memory per agent, and a KV cache offload strategy. The GPU is a resource the agent rents. The CPU is where it lives.

One thing to watch: the CPU:GPU ratio is moving from 1: