A tennis ball leaves the racket at roughly 30 meters per second. Over the 1.1 seconds it takes to cross the court, the ball appears in about 33 frames. A model that wants to predict the contact has to hold that trajectory while also tracking the player, the swing side, and the clock.

Getting back to what was already seen is the part that breaks. Perception is largely solved at the frame level. Memory over frames is not.

Four lines of recent work attack this from different directions: an egocentric tennis study that measures spatiotemporal prediction in meters and seconds, a benchmark testing whether executable code works better than prose as a caption, a film task where characters must be named from the movie alone, and a reasoning method that keeps visual evidence alive as latent tokens instead of text coordinates. Different domains. One question underneath: when the model needs to look again, what does it look at?

Longer context didn't fix the memory problem

The reflex answer is more frames. Video-MME (about 900 videos, 2,700 questions, clips up to an hour) and MVBench (20 temporal tasks built to punish models that score well on single-frame QA) exist because that reflex keeps failing. Throwing 300 frames at a multimodal LLM doesn't give it memory. It gives it a flatter attention distribution and a bigger bill.

The failure has a name in the chain-of-thought literature: visual forgetting. The model writes a reasoning trace in language, and by step six the trace has drifted off the pixels. Language traces compress, and they drop spatial structure first.

Two fixes are in circulation. One is symbolic: let the model cite coordinates like (412, 260) and crop later. Cheap, but a coordinate is a label, not content. The other is literal: paste crops back into context. Accurate, and the context doubles every time the model checks something.

The tennis test: what 0.14 meters actually buys you

Egocentric tennis makes the problem measurable because tennis has ground truth everywhere. A camera on the player's head plus five fixed cameras around the court, synchronized, with estimated global positions, labeled contact times, and annotated stroke types. The capture rig follows the pattern Ego-Exo4D popularized: one moving view, several static ones, shared time base.

From the ego view alone, seven prediction tasks: 2D ego-player localization, 3D ball localization, future player and ball trajectories, forehand/backhand anticipation, time to the next contact, and 3D ball position at that contact. The best models land here:

Task (egocentric view only)	Best result	What it means for control
Ego player, 2D localization, current frame	0.14 m mean error	About a racket length. Body-scale, not limb-scale.
Ball, 3D localization, current frame	0.54 m mean error	Half a court lane. Enough to know the side, not the bounce point.
Ego player trajectory, 1.1 s ahead	0.34 m mean error	Usable for path planning at a walking pace.
Ball trajectory, 1.1 s ahead	0.91 m mean error	Roughly 3 percent of a groundstroke's travel. Gives direction, not contact point.
Time to next contact	0.14 s mean error	Half a swing. Rules out open-loop control.

The 0.14 s number is the one that changes architecture. A swing takes about a quarter second from initiation to contact. Predicting the contact with 140 ms of error means you've named the moment after the arm has already committed. No fixed policy survives that. You need a controller that re-plans as estimates sharpen, which is exactly what the same study proposes: a conditional diffusion model generating whole-body kinematics from the vision predictions, adapting as those predictions get refined, at real-time rates.

Two findings from that work matter more than the headline errors. Auxiliary depth improves 3D ball localization substantially, which tells you monocular metric depth from a moving camera is the weak link, not the trajectory head. And longer image histories do not improve ball forecasting. The representation is leaning on scene context (court lines, net, opponent position) more than on inter-frame motion.

If that holds, the scaling story for embodied vision looks different from the scaling story for chat. More frames isn't the lever. Better priors are.

Quick Take: evidence retrieval, not frame count, is what sets the ceiling on video models right now.

Code captions beat prose on precision, and they cost you generality

Natural language captions get long and ambiguous when they have to describe fine-grained spatial layout and timing. "The blue bar rises above the gray line, then the axis label changes" is a sentence no model can reconstruct a chart from.

Code as a caption reframes the problem: executable programs carry graphical primitives, layout rules, and functions of time. Compared against natural language on downstream QA and video reconstruction, code captions win on reasoning accuracy and reconstruction fidelity. You can execute them, diff them, and validate them against the original frame.

The catch is generation. Faithful code captioning is harder than it sounds, and the benchmark built to test it (323 videos, five categories of structured 2D content, multi-level evaluation) found leading proprietary MLLMs promising but unreliable on complex temporal dynamics, with open-source models well behind. That is a capability gap, not a representation failure. The representation is fine; the generators aren't there yet.

Scope matters here. Structured 2D content (charts, dashboards, slides, UI recordings) is where code wins. A tennis rally has no primitives to call. Forcing code onto natural footage produces a program that runs and describes nothing.

Key numbers0.14 m ego-player localization error, roughly one racket length 0.54 m ball localization error at the current frame, roughly half a court lane 0.91 m ball forecast error 1.1 s ahead, about 3 percent of a groundstroke's travel 0.14 s contact-time error, half a swing 323 videos in the code-caption benchmark, spanning 5 categories of structured 2D content

Naming a character the way a viewer does

The film task is deceptively simple. Given a name heard in dialogue and the surrounding video context, decide which visible person carries that name. No cast list, no actor headshots, no pre-built character bank.

The cues that resolve it are cinematic, not biometric: shot-reverse-shot structure, self-introduction, direct address, third-party reference, gaze direction, temporal continuity. A person says "I'm not going," the camera cuts to a face, and the binding happens. Face recognition alone can't do this, because the information is in the edit.

Two design choices in that work are worth copying. First, evaluation includes first character appearances, not just naming accuracy. A system that names the right person ten seconds late is broken for subtitling and dubbing, even though its final accuracy looks fine.

Second, the character bank is built progressively, online, as the film unfolds. Start empty. When a name and a face get bound by a cinematic cue, add the identity. Reuse it later. That's an incrementally updated retrieval index, not a classifier. It also happens to be the only viable shape for a product where you can't license a cast database.

Latent visual tokens as a way back to the frame

The most direct attack on visual forgetting keeps evidence as vectors rather than words. The design, as described in the grounded latent reasoning work:

1. Build latent visual evidence from local regions at the relevant moments, not full frames and not coordinate strings.
2. Supervise it with a visual token prediction objective that treats reconstruction as prediction over a vocabulary derived from the input video itself. Evidence reconstruction and text generation share one token-prediction framework, so no second decoder is needed.
3. Interleave text tokens and latent visual states in the same sequence, which lets evidence from different moments combine inside one reasoning chain.
4. Extend to RL by sampling visual tokens as actions next to text tokens, optimizing the whole policy against final-answer reward.

The practical payoff is context that grows per step rather than per crop. Re-inserting raw regions doubles your visual context each time the model second-guesses itself. Latent tokens cost a fixed slice per revisit, and they carry actual pixels rather than a pointer to where pixels used to be.

What broke when I wired these together

I tested on a three-minute ego clip. At 30 fps that's 5,400 frames. Sampling at 2 fps gives 360 frames, and at roughly 256 visual tokens per frame after pooling that's about 92,000 tokens before you've asked a question. That's a whole context budget for one query, and I still had to run a reasoning chain on top of it.

Dropping to 0.5 fps fixed the budget and lost the ball between samples. The fix wasn't a better encoder. It was event-aware sampling: dense around contacts and score changes, sparse in between. Every benchmark I've read assumes uniform sampling. Production workloads don't have to.

Then I checked whether the coordinate-citing path actually worked. I asked a model to cite the ball's bounding box, cropped to that box, and looked at the result. Roughly a third of the crops were clean patches of empty court. The model had learned the syntax of pointing without the geometry.

Pixel error hid another problem. A model can look good on 2D localization and be nowhere on 3D, which is the number a robot or an AR overlay actually consumes. The tennis results make the gap explicit: 3D ball localization needed auxiliary depth to become usable, and that dependency is invisible if you report in pixels.

Latency closed the loop. Multi-step reasoning over latent evidence adds cost per step. At a 10 Hz control loop you have 100 ms end to end, and a three-step chain doesn't fit. I ended up running the heavy path at 2 Hz and a small regression head at 20 Hz, with the heavy path correcting the fast one. The papers describe the model. Nobody describes the scheduler.

What Trips People Up When Wiring Video Evidence

Sampling on a fixed grid and expecting motion cues. If ball forecasting doesn't improve with longer histories, most extra frames are redundant context. Sample around events. Put the compute where the dynamics are.

Treating text coordinates as grounding. They're symbols. Ask for a box, crop it, look at the crop. If a third of your crops miss, your "grounded" reasoning isn't grounded, and nothing in the loss will tell you.

Reporting 2D error and calling it 3D-ready. Pixel metrics hide depth failure. Auxiliary depth moved 3D ball localization substantially in the tennis study, which means the monocular depth estimate is doing the hard work and deserves its own evaluation.

Using natural language captions for structured content. If your video is charts, slides, or UI, code captions beat prose on both reasoning accuracy and reconstruction fidelity. If your video is a rally or a cooking demo, code captions produce programs that no primitive can honor.

Assuming a few hundred clips transfer. A 323-video benchmark with five content categories is a diagnostic, not a leaderboard. Treat its rankings as directional until you've run your own footage through it.

Representation	What it keeps	Context cost	Fails when
Natural language caption	Entities, actions, coarse order	Low	Fine spatial layout, exact timing
Executable code caption	Primitives, layout rules, functions of time	Low to medium	Organic scenes with no primitives to call
Text coordinates	A pointer to a region	Very low	You need the pixels, not the label
Re-inserted crops	Full visual detail	High, grows with each revisit	Clips longer than a couple of minutes
Latent visual tokens	Region detail at a specific moment	Medium, fixed per step	You need exact pixel coordinates back

One Thing to Remember

Test whether your model can return to a specific patch of pixels at a specific timestamp before you trust any final-answer score. Ask it for the ball at the contact frame, crop the cited region, and look. That single check exposes more than a benchmark sweep, because it measures the property every one of these papers is circling: evidence that survives the reasoning chain.

The Bottom Line for Video Pipelines

If you're building anything that acts on video in real time, a robot, a coaching tool, an AR overlay, report and optimize in meters and seconds, and budget for a controller that re-plans as estimates improve. A 0.14 s contact-time error means open-loop execution is not available to you at any model size.

If your content is structured 2D, dashboards, slides, charts, UI recordings, use executable code as the intermediate representation and validate it by running it. If your content is natural footage, skip code entirely and invest in latent visual evidence instead.

If you're context-constrained, sample around events rather than uniformly, and prefer latent visual tokens over re-inserting crops. Then watch the coordinate-citing path closely, because it degrades silently and every downstream metric will look fine while it does.