What Five New Benchmarks Reveal About MLLM Limitations

The press release says the model is the best yet. You paste in a video of a kitchen: a doorbell rings, a hand reaches for a mug, and you ask what happened first. The model says the hand moved before the doorbell. The flagship benchmark it topped said nothing about that failure, because it scored whole captions against whole captions, and the model writes fluent captions.

Five new papers, posted within days of each other, attack different slices of the same problem. OmniCapBench checks fine-grained audio-visual captioning. FastBench tests streaming video under tight context budgets. SpaceCast-Bench probes predictive spatial reasoning. GeoReform works through geometry diagrams. SiGMA rethinks industrial anomaly understanding. Different domains, same diagnosis: MLLMs perceive locally and integrate poorly. The shared method matters more: score atomic, checkable units instead of whole outputs, so every failure has an address.

Benchmark	Target capability	Test scale	Headline finding
OmniCapBench	Fine-grained audio-visual captioning	786 densely annotated videos	Strong local perception, weak long-horizon reasoning; identity drift and cross-modal misalignment dominate
FastBench	High-dynamic streaming video perception	306 QA pairs, 8 domains	Best model (Gemini-3.5-Flash) scores 50.7%; 2 FPS sampling misses the action
SpaceCast-Bench	Predictive spatial reasoning	3,862 questions, 182 scenes, 21 models	Best model 58.0% vs 87.2% human; specialized models near chance
Geometry3K + GeoReform	Multimodal geometry reasoning	~3,000 problems, 200-example diagnostic	Structure injection fixes 28 errors, causes 13; GeoReform lifts a 2B model from 42.0% to 56.0%
MMAD via SiGMA	Industrial anomaly understanding	Public MMAD benchmark	In-domain specialists top out at 75.5%; SiGMA reaches 85.2%, humans 92.3%

OmniCapBench: captions as checkable units

Caption evaluation has a structural trade-off that nobody likes to admit. Whole-caption scores give you coverage with no localization: you know the caption is wrong, not where. Local probes localize but sample so thinly they miss most of what a model does. Unconstrained LLM judges drift between runs, so your numbers move for reasons unrelated to the model.

OmniCapBench reframes the prediction target. Models output sets of atomic evaluation units across three tracks: entity references, visual shots, and audio events. A doorbell ring becomes an audio event with a timestamp. A person becomes an entity reference that must stay bound to the same identity. Whether the model placed the ring before or after the hand reach is now deterministically checkable, with LLM comparison limited to localized semantic judgments that actually need judgment.

The 786 densely annotated videos are enough to separate four error classes: temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Run frontier MLLMs against it and the pattern is consistent with everything else in this batch: solid local perception, weak long-horizon audio-visual reasoning. Models lose track of which speaker said what, and they misalign what they see with what they hear. With atomic units, you can point at the exact phrase, the exact shot, the second where it went wrong. That's the difference between a score and a diagnosis.

FastBench: the stream outruns your sampler

Streaming video models operate under a bounded context budget. Every frame you keep costs tokens, so the default answer is uniform sampling at 1 to 2 FPS. FastBench quantifies what that choice costs. Its 306 QA pairs are built from high-FPS clips, filtered to questions that are answerable at 2 FPS, verified with SAM3 and CoTracker3 trajectories, then put through three rounds of human inspection. The tasks span eight domains and forward, instant, and backward temporal scopes. These are fair questions about fast events, not tricks.

Then the numbers. Gemini-3.5-Flash, the strongest model tested, scores 50.7%. Qwen3-VL-8B goes from 32.9% at 2 FPS to 44.6% at 24 FPS, and the gains saturate as old frames get compressed out of the context window. Sampling density is a real lever, but you can't just crank FPS and keep the history.

The paper's training-free baseline, ProactiveFrame, splits the stream into two tiers:

A recent window keeps full-FPS tokens; older frames degrade into sparse history. This beats uniform sampling by 5.4 and 1.5 percentage points, yet stays well below an oracle that knows where the action will be. Translation: current VLMs can't decide from the stream itself when to look harder. If you deploy a streaming pipeline at fixed low FPS, no prompt engineering recovers the events you never sampled.

Quick take: Across all five papers, the result is consistent: models perceive locally, then break when asked to relate what they saw across time, space, or modalities.

SpaceCast-Bench: predicting what hasn't happened

Most spatial benchmarks test perception. They show an image and ask for relations that are already visible: the cup is on the table. Real spatial intelligence is predictive. You watch someone push the cup and you have to say where it lands, or whether the stack falls when the bottom block slides out. SpaceCast-Bench is built around an observe-transform-infer framework with 3,862 questions from 182 real-world scenes, spanning 16 task types at three levels: static perception, local prediction, global prediction. Each level adds load, from scene understanding to spatial state updating to relational inference over unobserved outcomes.

Across 21 models, the strongest reaches 58.0%. Human performance sits at 87.2%. Spatially specialized models are near random chance. Two controlled findings are worth stealing: bridge views, an extra view that connects distributed observations, are critical for integration, and explicit 3D evidence helps models more reliably than generated outcome images or videos. If you're building a spatial reasoning system, spend compute on geometry, not on image generation.

The capability is learnable, and it transfers. Fine-tuning on programmatically generated data lifts Qwen3-VL-4B from 34.0% to 65.7%, with macro-average gains across six out-of-domain benchmarks. The frontier models haven't learned it, but a 4B model can.

GeoReform: geometry needs policy, not extraction

Geometry diagrams are the purest test of multimodal binding. Models can read the text of a problem; using the diagram is another story. The popular fix is structure injection: convert entities, relations, and constraints into explicit text for the model to reason over. GeoReform's diagnostics show why that fix is unreliable. On Geometry3K, structure injection fixed 28 errors in 200 examples and introduced 13 new ones. Redundant relations distract the model, and ambiguous references to diagram elements make it apply constraints to the wrong object.

The paper's diagnosis: extraction was never the bottleneck. Organizing the facts is where performance lives. GeoReform treats formalization as an optimizable policy instead of a fixed parser output. It runs the full reasoning pipeline, collects failed rollouts, diagnoses defects in the current representation, and mutates the policy to better select, ground, group, and present entities, relations, constraints, and targets. On Geometry3K, that lifts Qwen3VL-2B from 42.0% to 56.0%.

A 14-point gain on a 2B model, from changing how information is presented rather than adding data or parameters. If you're building on small MLLMs, this is the cheapest lever in this entire batch.

SiGMA: in-domain training won't fix perception

Industrial anomaly understanding is where all four failure modes meet. A model must detect a defect, localize it, describe it, and reason about it. No single MLLM does all of that well. The result that should worry anyone fine-tuning: on MMAD, trained specialists reach at most 75.5% in defect localization against 92.3% for humans, and they detect anomalies less accurately than their own untrained base model. Training made them worse at detection.

Different MLLMs have complementary strengths but share the fine-grained perception weakness, so ensembling them alone doesn't help. SiGMA adds a spatial component and divides labor. A multimodal searcher supplies industrial knowledge and normal references, a defect expert turns query-reference comparison into calibrated anomaly evidence, and a label-free reliability controller weights each source by task-wise competence and query-level evidence quality.

The result: 85.2% average accuracy on MMAD, 4.0 points above the strongest trained specialist and Gemini-2.5-Pro, within 1.5 points of human experts. With three agents capped at 9B parameters, it still reaches 84.4%. Those are commodity-GPU numbers, no frontier model required. And new MLLMs join the framework without retraining, so the system improves as the model pool improves, while a trained specialist is frozen at its training time.

The pattern across all five

Put the numbers side by side and the story writes itself.

When I put these papers side by side, the same failure class keeps showing up: binding. Models struggle to bind entities to time (identity drift), vision to audio (cross-modal misalignment), and diagram objects to their labels (ambiguous references). The binding is the bottleneck, not local perception. SiGMA is the proof that the fix works: a defect expert that never has to be a generalist beats generalists at their own specialist task.

These benchmarks are also cheap to run, which is the unstated point. 786 videos, 306 QA pairs, 182 scenes, a 2B model. Once evaluation units are atomic, small test sets locate failures precisely enough that you don't need a million examples to find out a model is broken.

Key numbers

• 50.7% is the best FastBench score, posted by Gemini-3.5-Flash. Barely better than a coin flip on high-dynamic stream questions.
• 58.0% vs 87.2% is the best-model to human gap on SpaceCast-Bench across 21 models. A 29-point shortfall.
• 85.2% is SiGMA's average on MMAD, within 1.5 points of human experts (92.3%) and 4.0 above the strongest trained specialist.
• +14 points is what GeoReform's reflective formalization bought Qwen3VL-2B on Geometry3K, from 42.0% to 56.0%.
• 32.9% to 44.6% is the jump for Qwen3-VL-8B when FastBench sampling goes from 2 to 24 FPS, a measure of how much uniform sampling leaves on the table.

Common pitfalls

First, scoring whole outputs. I've run enough caption evals to know the trap: the model writes fluent, plausible text, the similarity score looks fine, and you ship it. Then a user points out the caption describes the wrong person speaking. Whole-output scores have nowhere to attach that failure. OmniCapBench's atomic units with deterministic constraint checks exist precisely because of this.

Second, fixed low-FPS sampling in streaming pipelines. A uniform 2 FPS sampler decides a priori that nothing important happens between frames. FastBench's 32.9% to 44.6% jump for Qwen3-VL-8B is the cost of that decision. But note the saturation: adding frames without an eviction policy just eats the context budget. Adaptive density, not raw FPS, is the design target.

Third, LLM judges without constraints. Free-form judge scores drift between runs and between judges, so benchmark deltas of a few points become noise. The fix that shows up across these papers: deterministic constraint checks for everything checkable, and LLM comparison only for semantics that genuinely need judgment, localized to the unit in question.

Fourth, assuming fine-tuning on the target benchmark closes perceptual gaps. MMAD's trained specialists detect anomalies worse than their untrained base. Fine-tuning optimizes the objective you give it, and if the model can't perceive the defect in the first place, that objective is unlearnable. Check whether your model can see the signal before you spend compute teaching it to talk about the signal.

Fifth, dumping every extracted fact into the prompt. GeoReform's 28 fixes and 13 regressions is the canonical example. Redundant relations add noise, ambiguous references misdirect constraints. Representation needs selection, grounding, and grouping. Treat it as a policy to optimize, not a parser to run.

The bottom line

If you're building evaluation for MLLMs, switch from whole-output similarity to atomic, verifiable units with deterministic checks. OmniCapBench and FastBench both show that this is what surfaces temporal grounding failures, identity drift, and cross-modal misalignment, the failures users actually hit in production.

If you're deploying streaming video pipelines under a context budget, treat sampling policy as a first-class design decision. Uniform 1 to 2 FPS forfeits fast events entirely, and ProactiveFrame's 5.4-point gain over uniform sampling costs no training. Expect adaptive frame-rate to become a standard evaluation axis within the next year.

If you're fine-tuning a small MLLM for a narrow domain like geometry or industrial inspection, put as much effort into input representation as into training data. GeoReform lifted a 2B model by 14 points by reorganizing how relations are presented, and SiGMA's sub-9B agents reached 84.4% by splitting perception into specialized roles. In-domain fine-tuning alone was the weakest lever in every paper in this batch.