Vision-language models that write radiology reports turn each chest X-ray into hundreds of small image patches, or "tokens," and the language model has to attend over all of them. Training-free token reduction methods drop or merge most of these tokens at inference time without changing any weights. On general benchmarks they look nearly free: up to 80–90% of tokens can go with only a few points of accuracy lost. Every one of those methods, however, has been scored on everyday photographs with question-answering accuracy. None has been scored with the clinical metrics radiology uses, which count whether a generated report keeps the findings that matter. In a chest X-ray the clinically decisive content is often a faint opacity or a thin pneumothorax line covering a few patches. Those patches are exactly the ones that attention- and redundancy-based pruning treats as background. Because report-writing models tend to fall back on fluent "normal" text when visual evidence is weak, pruning could silently turn abnormal studies into plausible normal reports, and surface-overlap metrics would not notice.
This note proposes two things that belong together. The first is a measurement: run existing plug-and-play reducers through radiology's clinical metrics, broken down by finding and by lesion size, to find out whether pruning causes silent omissions. The second is a training-free selection rule that uses the strongest prior specific to this domain. Chest radiographs are roughly spatially aligned, and most of the image looks like a normal chest. We therefore score each token by how far it sits from the features that normal studies produce at the same location, and we keep the deviant tokens plus a small set that covers the rest of the image. The hypothesis is that this "residual to a normal atlas" score keeps small abnormalities that attention scores miss. The method changes no model parameters. Its only extra ingredient is a feature statistic computed once from normal studies.
Core Claim
We make two claims. First, at the same token budget, existing training-free reducers lose much more on positive-finding recall than their general-benchmark retention suggests, and the loss is concentrated in small lesions. Clinical metrics such as RadGraph F1 and CheXpert F1-14 expose this loss, while BLEU and ROUGE largely hide it. Second, replacing attention or diversity scores with a deviation-from-normal score recovers most of that loss at aggressive budgets, for example 64 kept tokens. If the second claim fails, the first still stands as a negative result that practitioners need.
Where This Sits in Prior Work
Training-free reduction, the direction that supplies our methods and baselines, falls into three families. Attention-ranked dropping includes FastV, which drops the visual tokens receiving the least attention after layer 2 of the language model, and HiRED, which uses the vision encoder's summary-token attention to allocate and drop tokens across image tiles. Redundancy- or coverage-aware selection includes DivPrune, which picks a maximally spread-out subset in embedding space, SCOPE, which balances saliency against a coverage gain, and TokenNMS, which applies non-maximum suppression in two stages, first spatially and then by query relevance. Merging includes AdaptMerge, which clusters tokens and uses text relevance to guide the merge.
All of these are evaluated on MMBench, POPE, GQA, AI2D, and similar general benchmarks. The file itself flags their shared weakness: anything with near-zero initial attention is removed for good. That weakness motivates this work. We keep their plug-and-play setting and replace the scoring signal with one specific to the domain.
The clinical-evaluation direction supplies our measurement. Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation found that BLEU misses false predictions. It also proposed RadGraph F1 and RadCliQ, a composite clinical score, which correlates with radiologist error counts at Kendall's tau-b of 0.522. RadGraph F1 builds on RadGraph, a tool that extracts clinical entities and their relations from reports. We use these metrics as given and do not propose a new one. We add a finding-level "silent normalization" count, defined below, because averaged F1 can mask the rare-positive failures we expect.
LLaVA-TA constrains how we interpret results. It showed through a "blank image test" that report generators rely less on the image as the text grows, and that they hallucinate along narrative patterns. This is why we expect pruning damage to show up as fluent normal text rather than as broken text. It is also why we avoid pruning rules that use the text generated so far to choose tokens. LLaVA-TA itself retrains the model and adds a segmentation-mask input, so it increases visual input rather than reducing it.
The closest existing work combining the two directions is SLIM, which prunes whole-slide pathology tokens. It differs from this proposal in three ways. Its pruning module is learned. It is evaluated with classification and retrieval, not report fidelity. It degraded on the external EBRAINS set, a warning that we take seriously for our cross-hospital test. Two other papers make radiology models cheaper without reducing tokens. SLaVA-CXR shrinks the language model to 2.7B parameters and reports ROUGE-L and METEOR. DRRG masks text tokens during diffusion decoding. Both leave the visual token count untouched.
Method: Atlas-Residual Selection
We take a normal atlas from a few thousand MIMIC-CXR studies whose CheXpert labels are "No Finding." Each such image goes through the host model's frozen vision encoder, and the encoder output token at each grid position $j$ is stored. This yields a bank $B_j$ of normal features for each position. For a test token $i$ at grid position $p(i)$ with feature $f_i$, the residual score is its distance to the nearest normal feature within a small spatial window $W(p(i))$:
$$r_i = \min_{j \in W(p(i))} \; \min_{b \in B_j} \left(1 - \cos(f_i, b)\right).$$
The window absorbs misalignment from patient positioning, and the inner minimum makes the score nonparametric. Selection then greedily maximizes residual plus coverage. With kept set $S$, budget $K$, and trade-off $\lambda$, each step adds the token that maximizes $r_i + \lambda \min_{s \in S}(1-\cos(f_i,f_s))$. The second term is DivPrune's max-min distance, which keeps a sparse covering of normal anatomy so the model can still describe it. Kept tokens retain their original positional embeddings.
The budget can adapt. One variant keeps every token above a residual threshold calibrated on normal studies, which addresses the fixed-ratio limitation the file identifies. The cost is one nearest-neighbour lookup per token against a bank that can be compressed, for example by k-means centroids per position. We will report wall-clock time on real hardware, because the file notes that FLOP savings do not always become real speedups.
Research Program and Proposed Experiments
The host model is LLaVA-Rad, the report generator LLaVA-TA uses as its baseline. On MIMIC-CXR, LLaVA-TA reports unpruned LLaVA-Rad at RadGraph F1 29.4 and CheXpert F1-14 39.5. As a second host with a different vision encoder and language model, we will use SLaVA-CXR's CLIP ViT-L/14 with Phi-2 if its weights are available, to check that the conclusions do not depend on one model. We will test at kept-token fractions of about 50%, 25%, and 11%, the last matching the 576-to-64 setting used by SCOPE and TokenNMS.
Data come from three sources. The MIMIC-CXR test split is the main benchmark. The atlas uses only training-split normal studies. IU-Xray tests cross-hospital transfer with the MIMIC-built atlas held fixed. MS-CXR's expert bounding boxes let us bin positive findings by box area and ask whether loss concentrates in small lesions.
The metrics are RadGraph F1, RadCliQ, CheXpert F1-14 split into positive-class recall and specificity per finding, and BLEU-4 and ROUGE-L, which serve as the "insensitive" comparison. The silent-normalization rate is the fraction of studies whose unpruned report contains a CheXpert-positive finding that the pruned report drops or negates. We use this paired, per-study definition so that the measurement isolates the effect of pruning from the model's baseline errors.
The analysis has three parts. First, we plot clinical-metric retention against lexical-metric retention for every reducer and budget, to test whether lexical scores overstate safety. Second, we plot the recall drop against MS-CXR lesion area. Third, for a stratified sample of about 200 studies, we have a radiologist rate omissions in pruned versus unpruned reports using the error taxonomy from the evaluation-progress paper. This guards against labeler bias in the automatic metrics.
Baselines
The do-nothing reference is the unpruned host model, with LLaVA-TA's reported LLaVA-Rad figures of RadGraph F1 29.4 and CheXpert F1-14 39.5 as an anchor. We will rerun it ourselves so that every comparison uses one evaluation pipeline. A naive reduction reference uses random token dropping and uniform 2×2 average pooling at each budget. If either matches the sophisticated reducers on clinical metrics, then the "intelligent" selection in general-domain work buys nothing here.
The strongest existing competitors are DivPrune and TokenNMS, run as their papers describe. DivPrune is the most important. On POPE it lost only 3.4% F1 under aggressive pruning, while attention baselines lost 50–70%. Its max-min objective also naturally picks outlier tokens, which is close to what our residual is meant to do. TokenNMS reports 97.14% retention at 576-to-64 compression on LLaVA-NeXT. We found no reported result on Lacuna for any plug-and-play reducer on RadGraph F1, RadCliQ, or CheXpert F1-14, so we will produce those numbers ourselves. FastV and HiRED represent attention-ranked dropping, and SCOPE represents saliency plus coverage.
Every method runs on the same host, the same test split, and the same kept-token count. Each gets the same small hyperparameter search on a MIMIC validation split: thresholds for TokenNMS, the pruning layer for FastV, and $\lambda$ and window size for our method. We fix the budgets in advance and report the selection overhead in wall-clock time.
Two ablations isolate the key ingredient. The residual-only variant ($\lambda=0$) and the coverage-only variant, which is DivPrune itself, separate the contribution of the atlas from that of coverage. A shuffled-atlas control builds the bank from abnormal studies or ignores spatial position, to show that position-specific normality, and not just feature rarity, carries the signal.
Beating DivPrune means a statistically significant gain in positive-finding recall and RadGraph F1 at the 11% budget, measured with a paired bootstrap over studies, and a lower silent-normalization rate. Specificity must not fall by more than one point.
Why not just…?
Why not just downsample the X-ray to a lower resolution? FastV reports that, at matched compute, high-resolution input with pruning beats unpruned low-resolution input. The MIMIC-CXR paper warns that lossy spatial reduction can erase small findings such as pneumothorax. Both claims come from general images or anecdote, not from clinical-metric measurements. We therefore include resolution reduction at matched token count as an explicit arm. If it ties our method on small-lesion recall, the simpler option wins and we will say so.
Why not just use a smaller language model? SLaVA-CXR achieves a 5.3× speedup by shrinking the language model, and that is a legitimate alternative. The two approaches are complementary, though, because a smaller model still processes every visual token. Multi-view and longitudinal inputs such as MLRG's multiply that token count. We test our method on SLaVA-CXR itself, so the question becomes whether token reduction adds savings on top of a small model without clinical loss.
Why not just use DivPrune, since diversity already favours outlier tokens? This is the most serious threat. In a mostly uniform chest image, the max-min objective will pick some abnormal patches. Its definition of "unusual," however, is relative to the other tokens in the same image. A diffuse finding, such as bilateral effusion or a large opacity, is internally self-similar and can be represented by one token. A subtle finding near the textured hilum is not distant in feature space from its neighbours. The atlas residual measures unusualness relative to what normal chests look like at that location. The residual-only versus coverage-only ablation, together with the MS-CXR lesion-size stratification, settles this question directly. If DivPrune matches us across lesion sizes, the contribution reduces to the clinical audit.
Why not just report BLEU and ROUGE, which are easier to compute? The evaluation-progress paper shows that BLEU does not penalize fabricated findings. Our specific concern is pruning that produces fluent normal text, which lexical overlap with mostly-normal references would reward. The lexical-versus-clinical retention plot tests this directly rather than assuming it.
Decisive Checks
Three outcomes would decide the proposal. The audit claim holds if, at 64 tokens, DivPrune and TokenNMS keep more than 95% of BLEU and ROUGE but lose measurably on positive-finding recall, with the loss growing as MS-CXR box area shrinks. The method claim holds if atlas-residual selection closes most of that gap without a specificity cost. The method fails if the IU-Xray results show the MIMIC atlas transfers poorly, mirroring SLIM's EBRAINS degradation. In that case we would test whether rebuilding the atlas from a few hundred unlabeled target-hospital normals restores performance, which would still be a training-free fix.
References
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
- HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models
- DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models
- SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs
- Decoupling Spatial and Semantic Token Compression for Vision-Language Model Acceleration
- AdaptMerge: Inference Time Adaptive Visual and Language-Guided Token Merging for Efficient Large Multimodal Models
- Evaluating progress in automatic chest X-ray radiology report generation
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- Rethinking Radiology Report Generation: From Narrative Flow to Topic-Guided Findings
- Scalable Whole-Slide Vision-Language Modeling with Learned Token Pruning
- SLaVA-CXR: Small Language and Vision Assistant for Chest X-ray Report Automation
- DRRG: A Discrete Diffusion Framework for Radiology Report Generation