The Token Cost War Is Fought Across the Entire Serving Stack

The price of a token is set before the model runs

On September 10, DeepSeek shipped V4.1 Flash. Off-peak prices: 0.02 yuan per million tokens for cache-hit input, 1 yuan for cache-miss input, 4 yuan for output. Peak hours double those numbers. At 0.02 yuan, the input side is effectively free; the 4 yuan output price is where the bill lands. When a token costs fractions of a cent, the model's benchmark score matters less than its production cost.

The same week, Reuters reported DeepSeek hired CITIC Securities to prepare for a STAR Market IPO, and reports surfaced of a planned ~1GW datacenter in Ulanqab, Inner Mongolia, with at least 160,000 Huawei Ascend 950DT accelerators. At a quoted price above 250,000 yuan per chip, the chip bill alone exceeds 40 billion yuan, before land, power, and the building.

That's the shape of this phase of AI. Tokens get cheaper; the factories get more expensive. Competition has shifted from "who has the best model" to "who can manufacture a token for the lowest cost." That manufacturing cost is set across five layers: the datacenter, the hardware, the request scheduler, the serving stack, and the storage fabric. In the last two weeks, each of those layers moved.

Key numbers

• 0.02 yuan: DeepSeek V4.1 Flash off-peak price per million cache-hit input tokens
• 160,000+: Ascend 950DT chips planned for DeepSeek's Ulanqab datacenter (~1GW)
• 2.0x: throughput gain FlashVector found on Unity's model server
• 22M: requests per second OpenAI's Habitat storage platform handles
• <30%: average GPU utilization across Chinese AI compute centers, per Tencent Cloud's Zhang Jin

The datacenter: Ulanqab runs on geography and a different grid

DeepSeek posted a Ulanqab-based datacenter operations job in April, paying up to 30,000 yuan a month. The comments joked about taking the salary and watching stars over the grassland. The punchline is that location, not model architecture, is now the strategic variable.

Ulanqab sits 320km from Beijing, about 1.5 hours by high-speed rail. Two 144-core direct optical routes link it to the capital at roughly 2.1ms one-way, so the latency penalty versus hosting in the city is a few milliseconds, invisible to a user. The average annual temperature of 4.3°C buys close to ten months of natural cooling. But the actual moat is the Mongolia West Grid, run by Inner Mongolia Power Group, separate from the State Grid system that covers most of China.

That grid trades new energy seriously. In 2024, over 92% of market-traded power on it came from renewables, and compute-industry green power purchases reached 83% of consumption. Two multi-year power purchase agreements for big-data companies were filed at an average of 219.6 yuan/MWh, about 0.22 yuan/kWh. Delivered tariffs at some Ulanqab compute centers land around 0.358 yuan/kWh. The mechanism matters as much as the number: multi-year PPAs give operators a predictable energy cost they can design into a product, instead of a spot price they can only react to.

Then the arithmetic. A 1GW load running flat out consumes about 8.76 TWh a year. Every 0.01 yuan/kWh off the power price saves close to 90 million yuan annually; at 0.05 yuan, savings top 400 million yuan. That's why operators now build wind, solar, and storage next to the racks. Zhongjin Data's zero-carbon base pairs 300MW of new energy with 45MW of storage and feeds the compute center through a dedicated substation, covering about 70% of consumption from local green power. Inner Mongolia is formalizing the pattern as "incremental distribution grid plus green power direct supply," which is a long name for a simple idea: don't send the electrons on a tour of the province before they hit the GPU.

The danger sits next to the cheap power. Tencent Cloud's Zhang Jin put average GPU utilization across Chinese AI compute centers below 30%. Cheap electricity buys nothing when the silicon idles, because the capital cost of idle hardware dwarfs the power bill. Ulanqab now has 12.5GW of committed datacenter capacity, up from 3.3GW a year ago, making it one of Asia-Pacific's largest and fastest-growing AI compute markets per Goldman Sachs. But of the 7 million signed standard racks, only about 650,000 are built, and the city reports 172,000 P of operating compute. Land quotas are already slowing projects, and a local official told reporters the industry is still in its investment phase, "still losing money." Cheap tokens require the GPU to run, and the GPU to run requires orders.

Hardware: Apple is looking at the inference rack again

The Information reports Apple is considering an AI server built around M8-series chips, with discussions about incorporating Nvidia networking hardware. The option under consideration is NVLink Fusion, which provides the hardware and software to exchange data between chips, and which Apple could use to link M8 processors. The target is inference: companies that want to run trained models on their own equipment. A release could come in 2029, and the report stresses the plan is not final.

The Reddit threads on the story follow a familiar shape. People point at Xserve, which Apple discontinued in 2011, and ask what changed. I found myself skeptical of the timeline, but the strategy makes sense. Training racks belong to Nvidia. Inference is becoming its own product category, with volume that didn't exist when Apple left the server business. The NVLink Fusion piece is the detail to watch: an M8 server that can talk to Nvidia GPUs in the same cluster solves the integration problem that sank earlier Apple server efforts. The real test, if the product ships, is whether enterprise buyers trust a vendor that abandoned the category twice.

Quick Take: Cheap tokens start with cheap electrons, but sub-30% GPU utilization converts cheap electrons into expensive idle silicon.

Scheduling: LYREO treats edge selection as a delayed-reward problem

Agentic AI services push LLM inference out to edge servers, where requests arrive at heterogeneous nodes with different compute and network profiles. Server selection is an online problem with two nasty properties. First, request latency depends on multi-stage execution, transmission, prefill, iteration-level decoding, and KV cache evolution, so static latency models miss what actually happens. Second, the quality of a scheduling decision is observable only after the request completes, which rules out immediate evaluation.

LYREO attacks both. It builds a cross-slot inference model that tracks each server's KV cache state as a memory-time consumption metric, then casts the long-term load-balancing objective as a Lyapunov drift constraint. Because rewards arrive late, it applies reward redistribution with sequence-based return prediction to convert delayed outcomes into learning signals for earlier decisions. In simulations, it consistently beats learning-based and heuristic baselines on end-to-end latency and load balance.

The KV cache angle is the piece people underweight. Prefill and decode have very different resource profiles, and the KV cache is the state that carries across decision slots. A scheduler that routes on static load will pile long-context requests onto servers already saturated in memory, even when their compute looks idle.

The serving stack: FlashVector proves the model server is the untapped layer

Unity's Vector advertising platform runs a serving hierarchy: GPU kernels at the bottom, the ML framework's computation graph above them, the model server (NVIDIA Triton's C++ codebase), and an on-demand feature transformation service written in Python. Each layer demands different expertise, and no individual has all of it. Prior agent work showed human-level results on standalone GPU kernel optimization. FlashVector extends that agent pattern across the rest of the stack.

On Unity's production workloads, FlashVector delivered up to 2x throughput and up to 1.98x latency improvement on the model server, roughly halving response time, plus up to 1.6x throughput on the feature store. Note where the wins came from. Mostly from Triton server internals and a Python feature service that had seen little optimization attention, not from kernels that were already heavily tuned. That's the practical lesson for anyone running a serving stack: the unglamorous layers between the GPU and the application hold the biggest remaining gains.

Storage: Habitat shows the state layer becomes the constraint

OpenAI's first post on Habitat describes how a Python library grew into a globally distributed storage platform serving 1 billion ChatGPT users at 22 million requests per second. That's roughly 1.9 trillion requests a day, the kind of load that breaks storage systems bolted on after the compute was scaled.

The structural point is that storage demand grows with user count, not model size. Conversation state, checkpoints, embeddings, and feature data compound as the user base grows. GPU fleets scale linearly; the state layer compounds. Habitat's evolution matters because it shows storage has to be designed as a distributed system in parallel with the inference path. It also connects back to LYREO: the KV cache is inference state, and how you store, move, and expire that state shapes latency and load balance at every layer above.

The five-layer picture

Layer	System	What changed	Reported result
Datacenter	Ulanqab buildout	12.5GW committed capacity, multi-year green PPAs, direct-supply grids	~0.22 yuan/kWh PPA average; ~0.358 yuan/kWh delivered
Hardware	Apple M8 AI server	Nvidia NVLink Fusion integration under discussion	TBD, target 2029
Scheduling	LYREO	KV-cache-aware cross-slot scheduling, Lyapunov load control	Lower latency, balanced load in simulation
Serving stack	FlashVector	Agentic optimization across kernels, graph, Triton, feature store	2.0x throughput, 1.98x latency, 1.6x feature throughput
Storage	Habitat	Python library to global distributed storage platform	1 billion users, 22M requests per second

Read the table as one mechanism: token revenue per request is collapsing, so cost per token has to collapse at every layer. Each layer also carries its own failure mode. Ulanqab's gap between committed and built capacity. Apple's unconfirmed timeline. LYREO's simulation-only evidence. FlashVector's single production deployment. Habitat's operational complexity. The direction is consistent; the delivery is uneven.

Common pitfalls

Optimizing only the GPU layer. If your profiling stops at kernels, you're leaving gains on the table. FlashVector's best results came from the Triton server and the Python feature store, not the kernels. Check server-level batching, queueing, and feature cache behavior before rewriting CUDA.

Chasing power price without utilization. A long-term PPA in a cheap-power region is a liability if your GPU utilization sits under 30%. The power discount only pays off when the silicon runs. Put utilization targets in the same spreadsheet as the power price, with the same level of commitment.

Scheduling without KV cache state. Prefill and decode consume resources differently, and the KV cache is the state that carries across slots. Routing on static server load alone will overload memory on some nodes while compute sits idle on others.

Treating scheduling feedback as immediate. Latency consequences appear only after the request completes. Naive online learners mis-attribute outcomes to the wrong decisions. You need delayed-reward handling like LYREO's reward redistribution, or your scheduler chases noise.

Under-provisioning the storage fabric. At 22M requests per second, storage fails before compute does. Design the state layer as a distributed system alongside the inference path, not as an afterthought. Habitat is essentially a case study in what it costs to retrofit scale.

One thing to remember

The cost of a token is mostly decided before the model runs. It's set by the power contract, the grid structure, the scheduler's KV awareness, the server batching configuration, and the storage fabric. The model itself is a shrinking fraction of the total bill, and the teams treating it that way are the ones winning the cost war.

The Bottom Line

If you run production LLM serving, move your optimization effort up the stack. Kernel-level work is largely mined out; FlashVector's results suggest the model server and feature store still hold up to 2x throughput for workloads that haven't been profiled there.

If you're planning GW-scale capacity, treat GPU utilization targets as hard gates, not aspirations. A 0.05 yuan power discount is worth over 400 million yuan a year at 1GW, but only when the GPU runs. Cheap power plus idle silicon is the most expensive combination in this industry.

If you're procuring inference hardware, keep one eye on Apple's server plans. The 2029 timeline is unconfirmed and the project may die, but inference is becoming a distinct hardware category. If the M8 server ships with NVLink Fusion, expect it to slot into existing Nvidia clusters and trigger a new round of enterprise inference procurement. If it doesn't, Nvidia's grip on the rack tightens further.