Your Video Model Sees the Ball. It Can't Tell You Which One Moved.

A shell game is three cups, one ball, and eight seconds of shuffling. Humans track it without effort. Video-language models do not.

On real shell-game footage, open models from 8B to 235B parameters all land within ten points of chance. That sweep includes thinking variants, an RL-trained tool agent, and three training-free tool agents. Ten points of chance means a 235B model, the sort that eats multiple GPUs and a serving budget, plays the game like a coin flip with extra steps.

The team behind Ask About the Right Moment went hunting for the cause. Their answer: these models see the ball fine. They lose track of which ball is which.

Binding is a vision problem. Maintenance is a bookkeeping problem.

The paper splits the job in two. Semantic identity binding is deciding, at one moment in time, that this region of pixels is the object the question is about. Temporal identity maintenance is keeping that decision attached across the next 300 frames.

Video-LMs handle the first part. Ask a single frame "is this the queen of hearts" and you'll get a reasonable answer. Ask across a clip and the binding silently drifts, because nothing in the architecture pins a token to a physical thing frame after frame. Every frame is a fresh interpretation.

The diagram is the entire contribution. Fix identity once, when the evidence is visible. Carry it afterwards with a tracker. Never ask the model to reinterpret the video during read-out, because read-out is where the drift happens.

Track, Then Bind, and why it needs no training

The method is called Track, Then Bind, and it runs on frozen weights from start to finish.

A promptable video tracker follows every candidate object in the clip and draws each one with a track number. Meanwhile a frozen vision-language model answers a static question about the identifying evidence, something like "which cup was covering the ball at the three-second mark" or "what does the queen's face look like." That single answer binds the target to a track ID. From there the answer is computed in text from the track table. How many times was the mug covered? Count the frames in the track where a hand overlaps it. Where did the ball end up? Read the track's last position. No further vision.

The practical dividend is that one configuration covers all benchmarks. There's no per-dataset prompt tuning, no fine-tuned head, no benchmark-specific trackers. If you're shipping a video agent, that single config matters more than any individual score, because it means the pipeline doesn't fork.

I've watched teams try the obvious shortcut: draw the track boxes onto the video frames and hand the annotated video back to the VLM with the original question. It feels like it should work. On real cup games it falls 12 to 19 points below the agent. The model gets distracted by the overlays and re-litigates identity instead of reading the annotation.

Quick Take: Scaling video-language models from 8B to 235B parameters doesn't move object-identity accuracy out of the noise, because the bottleneck is temporal bookkeeping, not visual recognition.

The numbers, and what they cost you

Task	Track, Then Bind	Comparison
VET-Bench, both tasks	solved with one config and Qwen3-VL-32B	fine-tuned trajectory model reaches 92% and 90%; the same VLM voting over 16 samples sits at 30-32%
Perception Test, hardest cup games	55.4%	best published model at 41.5%
TOC-Bench	48.6%	same Qwen3-VL-32B answering directly at 37.1%
Real cup games, tracks drawn on video and asked directly	12 to 19 points below the agent	the agent is the reference

Read the VET-Bench row twice. A fine-tuned trajectory model hits 92% and 90%, which is the ceiling. The interesting comparison is the middle column: the same 32B model with 16-sample self-consistency voting gets 30-32%. That's sixteen forward passes, sixteen chances to be lucky, and it's still 60 points off. Voting fixes sampling noise. It cannot fix an identity that was never bound.

The TOC-Bench gap is the cleanest evidence. Same model, same questions, same video. The only change is whether the answer is derived from a track table or asked directly. Eleven and a half points separate those two worlds.

PointerBench measures the click you can't quite place

Video identity is half the grounding problem. The other half is where you click.

PointerBench is a diagnostic suite of 1,500 screenshot-instruction pairs, each labeled with pixel-exact target geometry. It splits into three parts: PointerBench-Sheets for thin sub-element identification and geometric grounding in spreadsheets, PointerBench-Text for character-level grounding in long documents, and PointerBench-Pro for mixed GUI grounding across 100 professional applications.

The authors evaluated general-purpose multimodal models plus 39 specialist GUI grounding configurations. Text grounding is where everything falls apart.

Claude Fable 5 scores 83.2% averaged across the suite and 69.6% on text. GPT-5.6 Sol goes from 77.7% to 53.4%. The leading native specialist, UI-Venus-2-9B, drops from 54.5% to 25.8%. That last number is the one that should worry anyone shipping a document agent. A specialist trained specifically for UI grounding succeeds on roughly one in four character-level document targets. At that rate you cannot let it click unattended.

Where the headroom actually is

Key Numbers

• 10 points: the maximum distance from chance achieved by any open video model from 8B to 235B on real shell games.
• 55.4% vs 41.5%: Track, Then Bind against the best published model on the hardest Perception Test cup games.
• 48.6% vs 37.1%: track-derived timelines against the identical model answering TOC-Bench directly.
• 1,500: screenshot-instruction pairs in PointerBench, each with pixel-exact target geometry.
• 39: specialist GUI grounding configurations evaluated, none of which clears 60% on text.
• 25.8%: UI-Venus-2-9B's success rate on PointerBench-Text, against a 54.5% suite average.

The shape of both results points the same direction. Aggregate benchmarks hide the failure mode that kills deployments. A model averaging 83% looks ready. A model scoring 69.6% on character-level document grounding does not, and the average was never going to tell you that.

Consider what 25.8% on PointerBench-Text means for a computer-use agent doing expense reports. Three out of four clicks land on the wrong character. The agent doesn't crash. It just edits the wrong cell and moves on, and you find out three weeks later when the reconciliation fails.

Hands-on notes from wiring this up

I built a small version of the tracking pipeline for a demo last quarter, before either paper existed. My first instinct was to annotate: run the tracker, draw numbered boxes on every frame, feed the annotated clip and the question to the VLM. It produced confident, wrong answers on every shuffle. When I read the 12 to 19 point gap, I recognized the failure immediately. The model was reading the overlay as content and second-guessing the numbering.

Switching to text read-out was the fix. Ask the VLM one static question at a high-evidence frame, get a track number, then compute everything downstream in Python. The VLM stopped being a video reasoner and became a labeler, which is a job it's actually good at.

The runtime surprise was that the tracker, not the VLM, dominated wall-clock on 60-second clips. A dense promptable tracker runs per frame and its cost scales with clip length, while the VLM ran exactly once. If you're budgeting latency, profile the tracker first.

On the GUI side, I've watched a computer-use agent try to fill a spreadsheet and click the row header instead of the cell. It was aiming at the right region and missing by one sub-element. That's the exact failure PointerBench-Sheets was built to isolate, and it's invisible in any end-to-end task-completion benchmark, because the agent recovers and the task still finishes.

Common Pitfalls

Re-reading the video at answer time. Don't hand annotated frames back to the model with the original question. Derive the answer from the track table in text. Drawing tracks on video and asking directly costs 12 to 19 points on real cup games.

Re-binding identity every N frames. Binding is only cheap because you do it once. If you re-ask the VLM "which one is the target" every few seconds, you pay full VLM latency repeatedly and reintroduce exactly the drift you were trying to remove. Bind at the frame with the clearest evidence, then trust the tracker.

Trusting average GUI scores. A suite-level number around 83% can coexist with 69.6% on text and far worse on spreadsheet sub-elements. Slice your evaluation by task type before you decide a model is deployable. PointerBench exists because the aggregates were hiding the failure.

Fine-tuning before you've tried tracking. A fine-tuned trajectory model reaches 92% and 90% on VET-Bench, which is excellent if VET-Bench is your product. A training-free agent with one configuration answers 55.4% of the hardest Perception Test cup games and 48.6% of TOC-Bench. Run the cheap version first and find out where it actually breaks.

Resizing screenshots to a square before grounding. PointerBench targets are pixel-exact. Thin spreadsheet rows and individual characters collapse when you downsample a 4K screenshot to 1024x1024. Ground in native resolution, or crop to a region of interest first.

Sources and code

Both papers are on arXiv. The video identity work is Ask About the Right Moment: Training-Free Object Identity Reasoning for Video-Language Models. The GUI benchmark is PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding, with the benchmark and evaluation code released at github.com/warmwindOS/pointerbench.

One Thing to Remember

Both papers say the same thing in different vocabularies: the hard part of grounding is holding a reference, not perceiving a feature. A model that can describe the queen of hearts perfectly is useless if it can't tell you she moved to the left cup. A model that recognizes a spreadsheet cell is useless if it clicks the label next to it. Reference-holding is a systems problem, and it responds to systems fixes like trackers and coordinate-exact evaluation, not to parameter counts.

The Bottom Line

• If you're building a video agent that answers questions about specific objects, put a promptable tracker in front of your VLM and keep read-out in text. The numbers are unambiguous: 55.4% against 41.5% on the hardest cup games, 48.6% against 37.1% on TOC-Bench, with frozen weights and a single configuration. Don't fine-tune until you've measured the training-free version.
• If your product touches documents, spreadsheets, or dense professional UIs, evaluate on PointerBench-Text and PointerBench-Sheets before you ship. A suite average near 83% can hide a 25.8% text-grounding rate, and a click that lands one character off is worse than a refusal because it fails silently.
• One thing to watch: the 12 to 19 point penalty for answering from annotated video suggests vision-side read-out is actively harmful, and I expect annotation-free prompting interfaces to become the default over the next six months. If your current pipeline passes frames plus overlays to the model, that's the piece to replace first.