Why a bigger inference budget stops buying accuracy

There's a specific kind of visual question that no amount of thinking fixes. Read the y-axis value at the third tick. Count the windows on the left face of the building. Confirm the total on the checkout page after the coupon applied.

The OpenVLHarness team ran a controlled study across 22 benchmarks and found the same thing I keep hitting in my own pipelines. Scaling the base model's inference budget produces modest gains. Not zero, but nowhere near the shape of curve you'd need if the answer requires a measurement that simply isn't in the pixels you were given.

The reason is structural. The model has perception, priors, and language. It does not have a ruler, a live web page, or a way to run arithmetic over an OCR result and a database lookup in the same breath. Tasks like that need grounded perception, external knowledge, and numeric measurement pulled through heterogeneous tools. Then the outputs have to survive long enough to be composed.

That's three separate engineering problems, and the three papers in this cluster each pick one.

Inside OpenVLHarness

OpenVLHarness is the most infrastructure-flavored of the three. It's a reasoning harness built around two pieces: a capability layer and a persistent multimodal memory.

The capability layer does two jobs. It gives the model task-level tool interfaces, so calling an OCR backend and calling a web search look the same from the model's side. And it does evidence rendering, which is the part most people skip. Raw tool output gets converted into something the model can actually read as an observation.

Persistent multimodal memory is the second piece. Images, structured data, and web references get retained as addressable artifacts for the life of the query. A coding interface then runs generated programs over that stored data, which is what lets you combine results from several tools without shoving everything back into the context window as prose.

The loop is acquire, inspect, compose. Nothing exotic. What makes it work is that the memory is addressable, so the model can point at artifact #7 instead of re-describing it, and the coding interface sits on top of that memory rather than on top of the chat history.

The numbers hold across eight backbones

This is the part that made me take the paper seriously. A harness that only helps one model family is a prompt trick. OpenVLHarness was evaluated on 22 benchmarks with eight open and proprietary backbones, and the average gain over matched no-tool baselines landed between 9.7 and 12.6 points.

Eleven points on an average benchmark score is the difference between a system you demo and a system you deploy. The cost side is more interesting still. With GPT-6 Luna at medium reasoning effort, OpenVLHarness beats the base model, Codex, and Visual Sketchpad at their maximum effort by 7.7 to 8.5 points while spending at most 37% of their model-token cost.

Read that again with the price tag attached. You get a better answer for roughly a third of the tokens, because the model stops burning context on reasoning it can't complete and starts spending calls on tools that can. That's the trade the paper is really selling.

Quick Take: raising inference budget mostly buys you longer wrong answers, while a harness with rendered tool evidence and persistent memory buys 9.7 to 12.6 points at a third of the token cost.

The negative result alone justifies a look at the OpenVLHarness study. Most agent papers show a gain and move on. This one measures how little you get from the obvious knob first, which tells you where the headroom actually lives.

Privileged-correction contamination

The second paper starts from a problem that only shows up once you've tried dense supervision. RLVR gives you reliable task-level rewards, but a single scalar at the end of a rollout tells you almost nothing about which token went wrong in a visual reasoning trace.

On-policy self-distillation patches that gap. A privileged teacher with an evidence-centered view of the image hands out token-level corrections. The trouble is that a correction can come from three places: the visual evidence, the teacher's language prior, or the teacher's response style. Distill all of it and you're training the student on the teacher's verbal habits as much as on the pixels.

The authors call this privileged-correction contamination, and they define it precisely: a token-level update counts as contaminated when its direction can't be consistently attributed to the privileged visual evidence. That's a clean formalization, and it explains a failure mode I've watched happen. The student learns to write confident-sounding reasoning steps that don't reference anything visible in the image.

Gating corrections by visual contrast

Their fix, Visual-Attribution Gating, is a validation step rather than a new loss term. The verifier-derived advantage from RLVR becomes the optimization anchor. Each privileged correction then gets checked against the contrast between the evidence-present teacher view and an evidence-degraded view.

If a correction points the same way whether or not the teacher can see the evidence, it isn't visual. Gate it out. If it flips when the evidence degrades, you have attribution, and you admit it.

Across six visual reasoning benchmarks, VAG beats both GRPO and OPSD on average. The design choice I find convincing is the anchor. Filtering dense corrections by consistency alone would be fragile, because the filter itself is a learned thing with its own failure modes. Anchoring to the verifier advantage means the task-level signal constrains what the dense signal is allowed to say.

Verification is a policy, not a prompt

VeriCUA is about the other end of the pipeline: after the GUI agent finishes clicking, how do you know it worked?

Recorded screenshots can't answer that. The confirmation dialog rendered in a toast that disappeared. The cart total changed only after a network round trip. So you need a proactive reward agent that pokes the live post-execution GUI and checks. Which raises two coupled decisions: what to check, and when to stop.

The default answer is to prompt a foundation model and hope. The VeriCUA authors measured how well that works. Prompted Judges from 4B all the way to GPT-5.4 plateau at 68% to 70% macro-average agreement. Bigger model: same plateau. Few-shot examples: same plateau. That flatness is the finding. Verification value depends on the environment state, not the task description, and it only becomes apparent through execution.

68% to 70%: the macro-average agreement band where every prompted Judge lands, from 4B up to GPT-5.4. 5.36 points: VeriCUA's agreement advantage over few-shot GPT-5.4. 47.6% to 59.8%: fewer Probe tokens than that few-shot baseline. 1B: the size at which a trained VeriCUA Judge still beats every prompted Judge. +3.21 / +5.40 points: task success gains overall and on held-out multi-application tasks.

Training it is awkward because the Actor's terminal state can't be faithfully replicated during training, so you get one verification attempt per state. VeriCUA handles that with a split architecture: a trainable Judge decides what to check and when to stop, and a frozen Probe executes the checks in the live environment. Online RL optimizes the Judge for verdict correctness and for efficient Probe use, with returns normalized across terminal states.

Trained only on OSWorld and evaluated with no further training on MobileWorld and WebArena-Verified, it lands on the agreement-cost Pareto frontier among proactive verifiers. Then it earns its keep downstream: used as the reward signal for Actor RL, it lifts task success by 3.21 points overall and 5.40 points on held-out multi-application tasks.

That 5.40 is the number to sit with. The gains concentrate on tasks the Actor never saw during training, which is where reward hacking usually bites.

Three papers, one table

	OpenVLHarness	VAG	VeriCUA
Problem attacked	acquiring verifiable evidence	token-level credit assignment	reward reliability after execution
Where the work happens	inference-time harness	RL fine-tuning	online RL on a Judge
Core mechanism	capability layer + addressable multimodal memory	gate corrections on evidence-present vs degraded contrast	trainable Judge + frozen Probe in the live GUI
Signal it trusts	rendered tool observations	verifier advantage as anchor	verdict correctness plus Probe efficiency
Scale story	8 backbones, 22 benchmarks	6 visual reasoning benchmarks	1B Judge beats prompted 4B to GPT-5.4
Cost story	37% of max-effort token cost	corrections only where attributable	47.6% to 59.8% fewer Probe tokens
Transfer evidence	broad backbone sweep	held-out benchmarks	OSWorld training only, zero-shot to MobileWorld and WebArena-Verified

The shared thread: none of the three wins by making the model bigger. They win by changing what the model is allowed to see, when it sees it, and what counts as proof.

Field notes from a half-built pipeline

I've been running a grounded-OCR-plus-browser agent for a few months, and these three failure modes are all familiar.

The tool-output-as-text problem bit me first. My OCR backend returned a number and I passed it through as plain text with no source frame attached. The model used a chart's axis label as a duration. Twice. Evidence rendering sounds like a formatting concern until you watch a model confidently mis-scope a measurement you handed it.

Unbounded memory bit me second. I kept every screenshot and every tool response in the transcript, and by step 12 the model was reading near-duplicate images that differed by one rendered frame. Retrieval got worse, not better, as the log grew. Addressable artifacts with stable keys fixed it. Volume didn't.

The verifier prompt plateau bit me third, and it's the one that stings. I spent a week upgrading my post-execution judge from a small model to a large one, then to a larger one with few-shot examples. My agreement with human labels barely moved off the high sixties. Reading VeriCUA was unpleasant in the way good papers usually are. I'd been tuning the wrong axis entirely.

Common pitfalls

Don't pass raw tool output to the model as prose. If an OCR result comes back as 4.7 with no frame, no units, and no pointer to the source artifact, the model will scope it to whatever it's already thinking about. Render evidence with its origin attached.

Don't keep memory without keys. Persistent multimodal memory only helps when artifacts are addressable. Dumping every observation into the context window gives you duplicates, not recall.

Don't distill every privileged correction. If the teacher's fix points the same direction with and without visual access, it came from a prior or a style, not from the image. Gate it.

Don't try to prompt your way out of a weak verifier. Four billion parameters and GPT-5.4 both plateau at 68% to 70% agreement. The check value lives in the environment, so it has to be learned from execution.

Don't treat verification as free. Proactive checking burns Probe tokens on every terminal state. Budget the check the way you budget the rollout, and normalize returns across terminal states so the Judge doesn't learn to give up early.

Sources

The three papers in this cluster: OpenVLHarness (arXiv:2610.26860), Visually Attributable On-Policy Self-Distillation (arXiv:2610.26856), and VeriCUA (arXiv:2610.26469).

One Thing to Remember

All three of these systems improve by changing the information flow around a fixed model, not by growing the model. Tool evidence rendering, attribution gating, and learned verification strategies are all cheap relative to another pretraining run. If you're stuck at a plateau, audit what your model can see and what counts as proof before you audit parameter counts.

The Bottom Line

If you're building a visual agent that needs measurements, live pages, or numeric composition across tools, add a harness with addressable multimodal memory before you touch the backbone. The 9.7 to 12.6 point average gain held across eight different models, and at medium reasoning effort you spend at most 37% of what a max-effort baseline burns.

If you're doing RLVR with dense self-distillation and your student is writing plausible reasoning that doesn't reference the image, suspect privileged-correction contamination. Gate corrections on evidence-present versus evidence-degraded contrast and anchor to the verifier advantage. Do not scale the teacher.

If your GUI agent's reward signal is a prompted judge, stop upgrading it. Every prompted Judge from 4B to GPT-5.4 sits in the same 68% to 70% band. Train a small Judge on single-attempt execution data, keep the Probe frozen, and expect the payoff to show up on held-out multi-application tasks, where VeriCUA added 5.40 points of task success.