Clinical Multimodal AI Can Name the Lesion. It Still Can't Point to It.

A multimodal model will tell you there's subretinal fluid in an OCT scan, then draw the box in the wrong quadrant. On RetinaGroundBench, the best general-purpose model out of 27 tested hits 73% balanced accuracy when asked whether a biomarker is present. Ask the same model to show you where, and macro F1 falls to 23.7%.

That 49-point gap is the most useful number in clinical multimodal AI right now.

The gap between naming a finding and locating it

RetinaGroundBench is the first benchmark built specifically for biomarker visual grounding in retinal OCT, the imaging modality ophthalmologists use more than any other. The scale is real: 815 B-scans, exhaustively annotated by ophthalmologists with 4,537 bounding boxes spanning 13 biomarkers across 8 diseases. Exhaustive matters here. Sparse annotation makes a grounding benchmark measure annotation coverage instead of model skill.

The authors also had to fix the metric. Multimodal LLMs tend to chop one lesion into several partial predictions, so per-box IoU punishes a model for being verbose rather than wrong. Their AggIoU measures whether the union of overlapping predicted boxes covers the annotated region. It's the right call, and anyone scoring MLLM output with stock IoU should copy it.

Then the results. Across 27 open-source models from 0.8B to 38B parameters, visual grounding is essentially unsolved. The best macro F1 is 23.7% with consistently low recall. A 23.7% F1 means roughly three out of every four lesion localizations miss the annotated region.

RetGrounder, their fine-tuned 8B model, gets to 33.9% for a 43% relative improvement. Two out of three localizations still fail. But an 8B model beat 38B general models, which tells you the bottleneck is task framing and domain data, not parameter count.

Five systems, one pattern

System	Task	Scale	Headline result
RetinaGroundBench / RetGrounder	Localize OCT biomarkers with boxes	815 B-scans, 4,537 boxes, 13 biomarkers	23.7% F1 general, 33.9% tuned
CellTrack-Bench	Link cells across frames	1,537 problems, 9,041 linking decisions, 15 datasets	22 VLMs benchmarked against specialized trackers
Lightweight Frame Compensator	Recover accuracy lost to frame skipping	3 polyp video benchmarks	2.05x-2.55x throughput, near frame-by-frame accuracy
Weak-supervision tumour localisation	Find HCC on intra-operative CT	External HCC cohort	Synthetic transfer beats CT-only baselines
Constrained health summarization	Turn structured records into patient narratives	4,500 summaries, 3 LLMs	Clinical alignment up, readability down

Read the rightmost column top to bottom. Every one of these systems wins by adding a small, task-specific component, not by scaling a general model.

Vision-language models versus the trackers they're meant to replace

CellTrack-Bench takes the question head-on: how do frontier VLMs do at cell tracking compared to the specialized algorithms already in production? The benchmark draws from 15 standard cell-tracking datasets across four imaging modalities and contains 1,537 problems covering 9,041 individual cell-linking decisions. Nine frontier closed-source models and 13 open-weight models were evaluated.

The construction detail I keep coming back to is the hard subset. The authors built it by finding failures shared across distinct algorithmic trackers, not failures unique to one. That distinction matters. A hard subset sampled from a single tracker's errors just tests that tracker's quirks. Shared failures across independent algorithms point at something structural in the problem.

Cell tracking is identity over time. You need to know the blob in frame 12 is the same cell as the blob in frame 8, and that it stayed the same cell through a division, a brightness shift, and a 4-micron drift. A model that reads frames as independent images has no mechanism for that. Nine thousand linking decisions spread over 1,537 problems works out to about six calls per problem, which means there's no error averaging. One bad link and the trajectory splits. Any tracking pipeline downstream of that is now tracking two cells that don't exist.

The paper's first contribution is evaluating environment harnesses for VLMs, and that's the right instinct. If a VLM can only succeed when the harness feeds it temporal context, then the harness is doing the work and the model is decoration.

Quick Take: Across five independent biomedical benchmarks, general-purpose multimodal models describe what they see and fail to show where it is, and the systems that close that gap are small, specialized, and trained on weak signals rather than expert labels.

77K parameters to buy back your frame rate

Real-time polyp segmentation during colonoscopy runs into a wall that has nothing to do with accuracy. The model is good enough. The latency budget isn't. Frame skipping is the standard workaround: run inference every K frames and reuse or interpolate the rest. Throughput goes up, and motion blur and lesion misalignment come along for the ride, which wrecks temporal stability. A polyp mask that flickers between frames is worse for a clinician than a lower-resolution mask that holds still.

The Lightweight Frame Compensator is 77K parameters. Insert it before a frozen segmentation backbone, give it a three-frame window, and it fuses the neighbours into one compensated frame via spatio-temporal attention. 77K parameters is under a tenth of a percent of a typical segmentation backbone, so you can drop it into an existing deployment without touching the weights you already validated or retraining the model that cleared regulatory review.

It trains on pseudo-labels from a differently seeded model. No expert annotation, which is the part that makes this deployable in practice. You can't ask a gastroenterologist to annotate compensated frames for a module that only exists because of your latency budget.

The throughput gain lands at 2.05x to 2.55x. If frame-by-frame inference gives you 12 fps, that's 25 to 31 fps, which is the difference between an assistive overlay a clinician will use and one they'll switch off.

The backbone analysis is the sharpest part of the paper. The authors ran three architectures: a CNN, a pyramid Transformer, and a global-attention Transformer. Sensitivity to frame skipping followed a spectrum, and that spectrum tracks how locally or globally each architecture aggregates misaligned input. Global attention pulls in the whole frame, so the blur from a skipped frame contaminates everything. The high-capacity Transformer backbones have the best static accuracy and suffer the most under skip-frame inference. The compensator's gain is largest exactly there.

That's a deployment diagnostic you can reuse. If your segmentation backbone has global attention and you're skipping frames, you already have this bug.

Key Numbers:

• 23.7% macro F1 for the best general-purpose MLLM on OCT grounding, versus 33.9% for an 8B tuned model
• 9,041 individual cell-linking decisions across CellTrack-Bench's 1,537 problems
• 77K parameters in the frame compensator, buying back most of a 2.05x-2.55x throughput gain
• 4,500 generated summaries across three LLMs, used to measure semantic drift and reading difficulty

Tumour localisation without paired scans

Liver HCC targeting during surgery has a data problem before it has a modeling problem. Pre-operative MR shows the tumour clearly and delineated. Intra-operative CT is where the surgeon actually is. Registering one to the other fails because tissue deforms when you open the patient, and paired pre/intra-operative datasets barely exist.

The approach here skips registration entirely. Train a classifier to detect tumour presence on pre-operative MR. Slide it over windows of the intra-operative CT. The domain gap is real, so the probabilities are noisy, but tumour-containing windows score higher on average. Those weak, noisy, cross-domain signals become the reward for a reinforcement learning agent that learns to localize tumours on the intra-operative image.

The comparison covers direct transfer, few-shot adaptation, and synthetic transfer, where CT gets translated into synthetic MR before localisation. On an external HCC cohort, synthetic transfer beat both the other transfer strategies and the CT-only baselines. External cohort is doing a lot of work in that sentence. A method that only wins on its own training distribution hasn't proven anything.

The general lesson: you don't need paired supervision. You need a signal that ranks better than chance and a policy that can exploit a noisy ranking. That bar is much lower than paired data, and insisting on paired data is a large part of why this problem sat unsolved.

The readability tax on patient-facing summaries

Wearables generate longitudinal physiological records that patients can't read. LLMs can turn those records into narrative, and in doing so they introduce semantic drift: clinically precise meaning gets distorted during simplification.

This study tests constrained generation on menstrual health data, combining structured evidence conditioning with retrieval-augmented generation. Three LLMs, 4,500 summaries, scored on NLI-based alignment, atomic factuality, and readability.

Grounding constraints consistently improved clinical alignment, particularly uncertainty preservation. Faithfulness measured against retrieved evidence generally held or improved. Omission effects depended on which model you used.

Then the cost: better clinical alignment came with increased reading difficulty and more medical terminology.

That tradeoff is the whole finding. The thing you do to make a summary safe makes it harder to read, and readability is the reason you built a patient-facing tool in the first place. Uncertainty preservation is the clearest example. "Your readings are consistent with X, though we can't rule out Y" is a clinically necessary sentence. Hedges are also the first thing simplification deletes, because hedges are what make text hard. Strip them and you get a confident sentence the data doesn't support.

What trips people up

1. Reporting presence accuracy in place of grounding accuracy. A model at 73% balanced accuracy on "is this biomarker here" and 23.7% F1 on "where is it" will pass a demo and fail a reading room. Score both, and report them separately.

2. Judging frame-skipping pipelines on static per-frame metrics. Misalignment between frames doesn't show up in single-frame IoU at all. If you don't measure temporal consistency, you'll ship a mask that flickers.

3. Scoring MLLM boxes with stock IoU. One lesion, three partial boxes, per-box IoU near zero. Use union coverage or you'll conclude the model failed when it merely fragmented.

4. Validating cross-modality transfer on the training distribution. The tumour localisation result means something because of the external cohort. Same architecture, internal split, no external validation, and you've measured nothing.

5. Tuning for faithfulness and shipping an unreadable summary. If you only report alignment metrics, you'll optimize the system toward clinical exactness with nine-syllable terminology, and patients will bounce off it. Report alignment, uncertainty preservation, and readability together or you're choosing a side in private.

One thing to remember

Every result in this cluster comes from a system that added a small specialized component to a general foundation: a grounding-tuned 8B head, a 77K-parameter frame module, a classifier repurposed as a reward signal, a constrained decoding wrapper. None of them won by making the general model bigger.

The bottom line

If you're shipping a clinical imaging assistant that reports findings, evaluate grounding before you ship. Presence accuracy around 73% will convince your reviewers, and localization F1 around 23.7% is what the radiologist will actually experience. Budget for the box metric, not the yes/no metric.

If you're latency-bound in a live procedure, colonoscopy or ultrasound or surgical navigation, don't swap backbones. Put a small compensation module in front of the frozen one and train it on pseudo-labels. 77K parameters at 2x throughput beats a new architecture you don't have time to validate.

One thing to watch: synthetic cross-modality transfer is about to make a batch of "we have no paired data" problems tractable. The CT-to-synthetic-MR trick beating direct transfer on an external HCC cohort is the proof of concept. Expect the same recipe copied into PET-to-CT, MR-to-ultrasound, and fundus-to-OCT pairs within a year, and expect most of those copies to skip the external validation that made this one credible.