The 100B phone target vs what ships today

Qualcomm's CEO says AI companies want phones running 100-billion-parameter models continuously by 2028. Translate that number into hardware reality: a 100B model at 4-bit weights is roughly 50 GB. A flagship phone today has 12-16 GB of RAM, and its NPU is designed for the 1-10B class. The 2028 target is a 3-4x memory gap and a bigger compute gap. Read it as a direction. The hardware to do it doesn't exist in phones today.

Meanwhile, this week's releases show what local inference looks like when engineers actually ship it: a 16.9 MB speech model that beats Whisper base on most benchmarks, a merged llama.cpp PR that adds 14% decode speed on prose, an Apache-2.0 clinical toolkit that runs fully offline, and an $18k, 128 GB VRAM server that carries a 10-person startup. The interesting work sits in that gap between the demo-video future and the stack you can deploy on Tuesday.

16.9 MB — Whistle's entire model, one file, CPU-only, no dependencies. Whisper base needs 145.3 MB for the same job. +14% — decode speedup on prose from probabilistic multi-token prediction in llama.cpp, now merged. 128 GB — total VRAM across 4x AMD R9700s, serving 16 concurrent sessions for a 10-person team. 0.07 s — time for OpenMed 3.0 to stitch five messy synthetic patient sources into one timeline, fully offline.

Whistle: speech recognition in 16.9 MB

Whistle is a speech-to-text model that fits in one 16.9 MB file and runs on the CPU with zero dependencies. It does three jobs, all on device: transcription of 16 kHz mono audio up to 30 seconds in one pass across seven languages (English, German, French, Spanish, Italian, Dutch, Polish, with auto-detection), per-word timestamps with probabilities, and speech embeddings from the encoder. It loads into the same C++ engine as Needle and ships prebuilt for seventeen targets: Android, iOS, watchOS, Windows on ARM, RISC-V, MIPS, the browser, WASI, plus the usual desktop platforms.

The front end is where the size savings start. Audio is framed at 25 ms windows with a 10 ms hop into 80 log-mel bins, band-limited to 250-3500 Hz, the telephone range. A convolutional stem of 128 channels halves that frame rate three times, so 30 seconds of audio becomes 375 frames at one per 80 ms. Everything after that runs at the slow rate.

The encoder is eight non-causal Simple Attention blocks with Monarch Hadamard MLPs in place of feed-forward networks, so a frame at 3 seconds can attend to a frame at 12 seconds. The decoder uses eight Laddered blocks at width 512, 8 query heads to 2 KV heads, 48-dim queries and keys, 64-dim values, and a 3-tap causal convolution on Q, K, and V. The speech-specific piece is one gated cross-attention per layer. Those projections run once when the clip arrives and stay cached for the whole decode, which is why five beams cost five short transcript caches, not five passes over the audio.

The practical result: time to first token tracks the clip length. 5.9 ms at 5 seconds of audio, 11.1 ms at 10, 36.3 ms at 30. Whisper pads every input to 30 seconds, so its latency is flat no matter what you feed it. And because every depth from 2 layers up was trained as its own model, --audio-depth trades quality for speed at load time without a separate download.

The benchmark reality check

Whistle's numbers were measured over 86,174 utterances and scored with the Whisper normalizers. No test audio appears in its training or validation data, verified by comparing audio checksums and speaker IDs across every reported test set. Whisper's and Moonshine's figures are the ones their authors published, from the multilingual checkpoints.

Dataset	Winner	What it measures
LibriSpeech test-clean	Whistle	Read speech, clean
LibriSpeech test-other	Whistle	Read speech, noisy
SPGISpeech	Whistle	Real-world financial audio
Earnings-22	Whistle	Earnings calls
FLEURS average	Whistle	Multilingual, the 7 supported languages
TED-LIUM	Whisper base	TED talks
AMI	Whisper base	Meetings
MLS average	Whisper base	Multilingual corpus

An 8.6x smaller model wins 5 of 8. That's the headline: size stopped being a proxy for quality in ASR. But the two Whisper wins on TED-LIUM and AMI are exactly the meeting-heavy domains, so if you're transcribing meetings, evaluate before you switch.

Quick take: the next round of local inference isn't about cramming a bigger model onto a device. It's about small models that beat the big ones on the tasks that matter, plus decoding tricks and serving stacks that make the whole loop cheap.

Faster decode without better hardware

llama.cpp merged PR #27694, probabilistic multi-token prediction (MTP). Instead of greedy drafting, the draft model proposes multiple token continuations with probabilities and the target model verifies them in one parallel pass. On prose generation, decode comes out about 14% faster. The tests ran with thinking off and ngram-mod off, and the optimal draft-n-max and draft-p-min settings land in line with greedy sampling.

What 14% means in a real workload: roughly one extra token accepted for every seven generated, with no measured quality cost. For long-form generation, RAG summaries, documentation, anything that emits paragraphs, that's the closest thing to a free lunch in decoding right now.

I updated llama and ran my usual summarization test on a 7B Q4 model. The gain showed up exactly where the PR thread said it would: prose. Code and structured output barely moved. I also found the trap: with thinking enabled, the draft distribution and the target distribution diverge and the acceptance rate drops. The 14% number assumes thinking off and ngram-mod off. Turn both off, set the draft parameters to match greedy, then measure your own workload before you trust any benchmark post.

The merged PR is in current llama.cpp. If you haven't pulled in a while, this is a good reason.

OpenMed 3.0: local AI that refuses to guess

OpenMed is an Apache-2.0 medical AI toolkit with one rule: patient data stays on the machine. No API keys, no cloud calls, works offline. 2,200+ compatible models on Hugging Face run through Transformers, ONNX, GGUF in your own llama.cpp, in the browser via WebGPU/WASM, or on a phone.

Version 3.0 adds Patient Journey, which stitches clinic notes, hospital FHIR feeds, HL7 messages, lab CSVs, and imaging reports into one timeline per patient. The design philosophy is simple: it refuses to guess.

• A 99% identity match? Not merged. A person reviews it, because a wrong merge puts one patient's allergies in someone else's chart.
• Two sources disagree on a diagnosis? Both stay, flagged for review.
• A lab corrected from 6.8 to 4.4? Both values are kept, and you can read the record exactly as it was before the fix.
• A note only says "January"? It won't invent January 1st to fill an OMOP column.
• You ask for a summary with a local model you haven't downloaded? It refuses. It never quietly calls a cloud API instead.

That last point is the whole philosophy. With clinical data, a silent fallback to a hosted model is a privacy decision nobody made.

I cloned the repo and ran the golden journey demo: five messy synthetic sources become one timeline in about 0.07 seconds. The maintainer is upfront about limits. The demo patient is synthetic, the summarization pipeline is synthetic-only in 3.0 and not ready for real patients, and the release doesn't ship new clinical checkpoints. This is plumbing. The clinical product comes later. But it's being built in the open: 148 PRs went into 3.0, 46 from the community, and 3.1 already has 422 open issues with 30 tagged for first-time contributors.

The 128 GB VRAM box that serves a startup

The piece most teams will actually buy is a build posted for a 10-person startup: $18k for a Threadripper 9970x, 128 GB DDR5 ECC, and 4x AMD Radeon R9700 AI Pro 32 GB cards, 128 GB VRAM total, running off a 1600W PSU with the GPUs undervolted under 210W each. A Radiance fork serves Qwen models to 16 concurrent sessions.

The numbers that matter, both Qwen configs with BF16 KV:

Model and config	Context per user	Prefill (aggregate tok/s)	Decode (aggregate tok/s)
Qwen 3.8 27B, MXFP4	4k-128k	6.3-6.8k	900 down to 80
Qwen 3.8 Next flash, int4fp8	48k	6,880	538.5
Qwen 3.8 Next flash, int4fp8	128k	3,050	365

Reading those numbers: at 48k context per user, aggregate prefill at ~6.9k tok/s ingests a 100-page document in about 7 seconds, and decode at 538 tok/s across 16 sessions means each user gets roughly 33 tok/s, which feels like a hosted API. At 128k per user, the aggregate decode on the 27B collapses to 80 tok/s, about 5 tok/s per user. That's readable streaming, not chat.

The lesson from the GPU split: VRAM for weights was never the hard part. Sixteen users at 128k context each means 2M tokens of live KV cache in BF16, and every decode step walks all of it. Budget KV cache before you promise long context, not after.

What trips people up

Benchmark ASR on size alone and you'll get burned in meetings. Whistle wins 5 of 8 against Whisper base at one-ninth the size, but TED-LIUM and AMI still go to Whisper. Meeting-heavy domains need their own eval. And remember Whisper pads to 30 seconds: its flat time-to-first-token hides the actual compute, so wall-clock comparisons on short clips will mislead you.

Running MTP with thinking enabled is the quickest way to conclude it's useless. The 14% gain was measured with thinking off and ngram-mod off. With thinking on, the draft distribution mismatches the target and acceptance drops. Keep draft settings aligned with greedy sampling, then measure on your own prose.

Planning VRAM for weights only is how you overpromise context. The R9700 box fits the weights easily; the KV cache is what dropped aggregate decode from 900 to 80 tok/s at 128k context per session. The same box that streams chat to 16 users at 48k context limps at 128k. Compute KV cost per user before you promise context length.

Silent fallbacks in privacy-sensitive tools are the liability no one planned. OpenMed refuses to summarize if the local model isn't downloaded. If you build clinical or legal tooling, audit your dependency chain for hidden telemetry or hosted-model fallbacks. A privacy decision nobody made is a liability you'll own.

Skipping quantization checks at long context compounds errors quietly. The build ran BF16 KV for every config. Mixing 4-bit weights with lower-precision KV at 128k context degrades both quality and throughput. Test the exact quant and context length you plan to ship.

One thing to remember

The pattern across all four releases is the same: local inference stopped being about squeezing weights into memory and became about engineering the whole loop, the audio front end, the acceptance rate of drafted tokens, the KV cache budget, and the refusal to leak data. The things that shipped this week win by being small, honest, and measurable, not by being the biggest model that fits.

What this means

If you're shipping speech recognition on phones, wearables, or embedded devices, start with Whistle. One 16.9 MB file, CPU-only, seven languages, and it beats Whisper base on 5 of 8 benchmarks. Verify against TED-LIUM and AMI before you commit to meeting-heavy domains, but the default should be the model that fits in a microcontroller's flash.

If you're generating long-form text through llama.cpp, update it and turn on probabilistic MTP. Fourteen percent faster prose decode with no quality loss at greedy-aligned settings is the best speedup-per-hour-of-work you'll get this quarter.

If you're buying one box for a small team, the R9700 build works, but plan KV cache before context length. Sixteen users at 48k context each stay chatty-fast at roughly 33 tok/s per user. Promising 128k per user drops that to about 5 tok/s on the 27B class. Sell the shorter context and deliver it reliably.

One thing to watch: the 100B-on-phone target for 2028 assumes memory and NPU bandwidth roughly triple. Expect the 2026-2027 phone generation to land at 30-50B class at 4-bit, which changes what "local" means for most products long before 2028 arrives.