The pitch for browser AI writes itself. No inference API. No network round trip. No per-token invoice. Data never leaves the device.

Then you open the same page on a phone.

I rebuilt five small AI features straight into a React and TypeScript app with Transformers.js 4.3.1. Web Workers keep inference off the main thread. WebGPU takes the heavy path where the browser exposes it, and WebAssembly picks up the rest. Zero backend anywhere in the loop. Four features held up in testing. Two of them timed out on a Samsung Galaxy S24 FE, which is a 2024 mid-flagship, not museum hardware.

That gap is the whole story of on-device AI right now. So here are the numbers behind it, and what they mean for the features you'd actually ship.

Five features, five very different byte budgets

Every feature uses a different model with a different job and a different download cost.

Feature	Model	Parameters	First-load download
Background removal	RMBG-1.4	~44M	~170 MB
Alt text generation	Florence-2-base-ft	~230M	350-450 MB
Structured ticket creation	Qwen2.5-0.5B-Instruct	~500M	~786 MB
Semantic search	all-MiniLM-L6-v2	~22M	23 MB (quantized CPU) / 90 MB (WebGPU)
Live visual Q&A	LFM2.5-VL-450M	~450M	350-400 MB

Parameter counts matter less than byte counts. A 22M-parameter embedding model landing at 23 MB means 8-bit quantized weights, and those run in a Web Worker on a mid-range laptop with no GPU at all. The 500M-parameter Qwen model at 786 MB is a different animal. Third of a gigabyte on a phone tethering over cellular, and the browser can evict that cache whenever storage pressure demands it.

23 MB is a rounding error next to a hero image. 786 MB is a decision the user has to make.

The alt text generator is the sleeper hit. Florence-2-base-ft at 230M parameters produces usable image descriptions, and for accessibility work that's genuinely valuable because a human still edits the result before publishing. The generated caption is a first draft, not the final copy, and good alt text depends on why the image is there, which the model doesn't know.

Quick Take: Browser AI is production-ready for narrow tasks that ship a small specialized model, and still a coin flip for anything that needs a 500M+ parameter language model on mobile.

The download is the design constraint

The chart below is the budget you're spending before a single token gets generated.

Browsers cache those files, so a returning visitor usually doesn't re-download anything. The first visit still hurts, and cache eviction isn't something you control. That's the argument for loading models on demand and treating every AI feature as optional. Nobody should wait on a 786 MB download to open your app.

Offline capability comes free once the files are cached, and so does the privacy story. Inference happens on the user's device, so sensitive text and photos never cross the wire. For support tickets and health-adjacent data, that's not a nice-to-have, it's the reason the feature exists.

A 22M embedding model outruns a 500M chat model at search

Semantic search was the feature that surprised me most. I indexed a batch of support tickets and searched them by meaning instead of keyword. The model is all-MiniLM-L6-v2, 22M parameters, 23 MB quantized, and it runs on CPU fast enough that you don't notice it.

The mechanism is plain. Convert each ticket and the query into a vector, then rank by cosine similarity. No generative model involved, no tokens billed, no hallucination surface. I kept a keyword search next to it for comparison, and the semantic results were better on paraphrased queries by a wide margin.

That's the lesson worth carrying into your own stack. A small, specialized model beats a general-purpose language model whenever the task has a fixed shape. Embeddings for retrieval, a segmentation model for images, a captioning model for alt text. The generalist only earns its 786 MB when the output has to be free-form and context-dependent.

I tried that generalist route anyway, and it cost me. For the ticket creator I assumed a language model would trivially turn a messy complaint into structured fields. It took far longer than expected. The hard part wasn't understanding the complaint, it was producing consistent JSON with the right fields and without invented details. I ended up with a carefully shaped prompt, an output parser, and Zod schema validation on top. The generated ticket stays editable before submission, which is the part that makes it safe to ship.

When cosine similarity calls a denial a confession

The same embedding machinery shows up in a game world where five llamas gossip with each other, built by a different author on top of llamadart and EmbeddingGemma 300M Q8_0. The goal is to decide whether one character revealed a secret to another. That's a retrieval problem on the surface and a judgment problem underneath.

I ran the companion example on macOS arm64 with the quantized EmbeddingGemma artifact, and it returned 768-dimensional vectors. Here's what the cosine scores looked like.

Input line	Compared against	Cosine similarity
"I saw Pip's scarf tangled in the reeds by the pond."	Scarf location fact	0.8449
"I did not knock the scarf into the pond."	Mo's accident fact	0.6085
"The sky looks beautiful today."	Scarf location fact	0.3137

Read the second row twice. The denial scores 0.6085 against the exact fact it denies. Embeddings encode topical similarity, and negation barely moves the vector. Mentioning the scarf and the pond is enough to land near the top of the ranking.

Scoring is a shortlist step. Correctness lives in a separate judgment, and the game routes shortlisted facts to a generative outcome model that has to decide whether something was actually said aloud. That model can be wrong too.

The fallback branch is the bug. A similarity threshold of 0.5 can accept a transfer even when the outcome model says nothing was claimed, and the keyword groups match "I did not knock the scarf into the pond" just as happily as a confession. I reproduced it directly: feed 0.6085 into the acceptance function and you get accepted=true with claimed=false.

The fix is structural. Drop the similarity-only fallback for state-changing events and require an explicit revelation judgment with supporting dialogue. Similarity can rank candidates. It shouldn't mutate knowledge.

The broader pattern is the one to steal: narrow candidates with application state, use embeddings to find plausible matches, and give anything consequential a separately evaluated decision. Then track provenance. Record who told whom, and keep "heard it" separate from "believes it."

WebGPU availability is the part that breaks

Everything above assumes the inference backend works. On desktop it usually does. Mobile is where the plan meets the device.

Two of my demos timed out on that Galaxy S24 FE. Not slow, not degraded. Dead, because WebGPU wasn't usable for those workloads and the WebAssembly CPU fallback couldn't finish in time. Some Transformers.js models fall back to CPU with SIMD or multithreading depending on the environment, and that can be an order of magnitude slower.

Key Numbers22M parameters / 23 MB for the quantized embedding model that runs without a GPU. 768 dimensions in every EmbeddingGemma 300M Q8_0 vector. 0.6085 cosine similarity between a denial and the fact it denies. 0.5 the similarity threshold that accepts that denial anyway. 786 MB the download cost of the one model whose output is hardest to constrain.

A feature review that stops at "WebGPU is supported in Chrome" is incomplete. Support depends on browser version, OS, GPU generation, and driver, and a powered GPU doesn't guarantee a fast one. I'd treat hardware capability detection as a first-class part of the feature, not an afterthought, and I'd design the degraded path before the fast one.

What trips people up

Letting a similarity score change state. A 0.5 threshold on cosine similarity accepts negations, as the 0.6085 denial shows. Use similarity to build a shortlist, then require an explicit judgment before anything commits.

Shipping a model without reading the license. RMBG-1.4 does excellent background removal, and the technical concept fits e-commerce perfectly. Its licensing conditions still apply, so check before commercial deployment. The same goes for any model you bundle.

Changing the input format after you've cached vectors. EmbeddingGemma expects task-specific prefixes, and the game used title: none | text: on both queries and documents. Change that string and every stored vector and threshold is invalid. Re-embed the corpus when the model or its formatting changes. Equal vector length across two models doesn't mean compatible representations either.

Trusting generated structure without validation. My first ticket creator returned malformed JSON and invented fields. Parse the output, validate it against a schema, and keep the result editable. A language model should draft, and a human should confirm.

Assuming every visitor gets the same experience. If WebGPU is missing and the CPU fallback can't hit your latency budget, the feature has to disappear cleanly. Test that path on a mid-range Android phone, not just your dev machine.

One thing to remember

The model you pick is a runtime decision, not just an accuracy decision. A 22M embedding model and a 500M chat model can both be "correct" on your test set and behave completely differently on a phone with no usable WebGPU, and only one of them will still work.

What this means

If you're building search, deduplication, or recommendation over a modest document set, ship a local embedding model and skip the chat model entirely. all-MiniLM-L6-v2 at 23 MB runs on CPU, returns results fast enough to feel instant, and keeps sensitive text on the device.

If your users are mostly on mobile or low-end hardware, don't build a feature that depends on a 500M+ parameter model loading successfully. Make it progressive, detect the backend, and fail to a simpler path rather than a spinner.

One thing to watch: WebGPU coverage on mobile is the variable that moves fastest here. Expect the two demos that timed out on a current mid-flagship to run on next year's devices, which means writing the capability check now rather than hard-coding assumptions about who can run what.