A chest X-ray at 1024×1024 pixels, cut into 16×16 patches, gives a vision transformer 4,096 tokens. Standard self-attention compares every token with every other one, so each layer's cost grows with the square of that count, roughly 16 million comparisons per layer. Most medical vision-language models avoid this cost by shrinking the image, which also erases the faint lines and small opacities that radiologists look for. General-domain methods save compute by discarding or merging image tokens. The best known, TRIPS, decides which patches to keep by asking the paired caption, and that choice has two flaws in radiology. At test time a diagnostic model usually sees the image with no report. And radiology reports are dominated by statements about what is absent: "no pneumothorax" names the very finding a naive text query would steer attention toward.
We propose to use the report only during pre-training, and only its positive findings, to teach the image encoder a small, text-free scorer. This scorer decides which high-resolution tokens stay separate and which get merged into their neighbors. At inference the encoder spends its token budget where findings tend to be, with no report needed. The bet is that at a fixed compute cost, the model sees a 1024-pixel image through this report-taught budget and beats both a full-attention model at lower resolution and a text-agnostic merging model at the same resolution. The gain should concentrate on small, focal findings.
Core Claim
Under matched encoder compute, radiology supervision about where positive findings are is a better signal for allocating visual tokens than either the image's own redundancy or whole-report text similarity. That supervision can be distilled into an image-only scorer during pre-training. This differs from the closest work in three specific ways. TRIPS uses the caption's summary token to prune 30–50% of patches, so it needs text at inference and was tested only on web images of everyday objects. Token Merging (ToMe) and DTMFormer merge tokens by visual similarity or attention alone, with no notion of clinical relevance. Medical vision-language models such as ASG, ECAMP, BERTHop and DermFM-Zero all process uncompressed patch grids or bypass the transformer's attention entirely. None of them treats resolution as a budget to spend.
The three input directions play different roles here. Domain-specific pre-training is the source of the signal. BERTHop shows that swapping general BERT for a clinical language model raised "Mass" AUC from 0.64 to 0.99, so we assume the finding embeddings must come from an in-domain text encoder. Quadratic attention is the constraint that makes allocation necessary. The cross-modal gap, meaning that image and text embeddings sit in separate regions of the shared space, is a correctness issue in the scorer and an evaluation lens. It is not a gap-closing objective we are selling.
Research Program
Components. The image encoder is a ViT-B that takes 1024×1024 radiographs. The text encoder is a clinical BERT, BlueBERT or PubMedBERT as used in BERTHop and DermFM-Zero.
Parsing reports. During pre-training on MIMIC-CXR (about 217,000 image-report pairs, per ASG), each report is parsed into finding entities marked present or absent. RadGraph-style parsing follows ASG. Entity-centered summaries follow ECAMP, which documents the more than 20:1 ratio of negative to positive descriptors.
Protection maps. Each positive entity is embedded by the text encoder and scored against early-layer patch tokens to produce a soft protection map. Negated entities are excluded. Reports with no positive findings produce a flat map, so the model falls back to similarity-based merging.
The scorer. A two-layer head on early visual tokens is trained to predict the protection map from the image alone.
Merging. A ToMe-style merge then combines the most similar unprotected tokens, which are mostly homogeneous lung, soft tissue and background. Protected tokens stay separate. Merging, rather than pruning, keeps diffuse texture present in aggregate. This matters because many chest findings are spread-out textural changes rather than compact objects, as BERTHop stresses. We keep ToMe's proportional attention, which gives merged tokens voting weight in proportion to how many patches they absorbed.
Dense tasks. For segmentation and detection, merged tokens are expanded back to the full grid using the recorded merge history, following DTMFormer's light reconstruction step.
Training objective. The base objective is contrastive image-report alignment, with the soft positives of ASG. ASG shows that hard negatives between patients who share a diagnosis penalize exactly what the model should learn.
Why the gap matters for the scorer. Mind the Gap shows that each encoder packs its embeddings into a narrow cone, and the two cones point in different directions. So image embeddings share a common offset $c$ relative to text. For a single finding query $q$, the score $q\cdot(v+c)$ equals $q\cdot v$ plus a constant, so ranking patches within one query is unaffected by the gap. When several findings are combined under one threshold, however, each query carries its own shift $q\cdot c$. Findings whose embedding happens to align with the gap direction would then capture most of the budget. We therefore build protection maps from per-query ranks rather than raw cosine scores, and we test that choice directly.
Gap as a measurement. We will measure how token merging during pre-training changes the gap distance and the grouping of patients by diagnosis. Closing the Modality Gap Aligns Group-Wise Semantics proves that retrieval depends only on rankings and is largely unaffected by the gap. Clustering by concept depends on distances and is gap-sensitive. The same paper reports that its gap-closing losses help less on heavily imbalanced label sets. Radiology is such a setting, so whether gap closing helps here is an open question we will test, not assume.
Baselines
The naive reference is a full-attention ViT-B at low resolution, pre-trained with the identical contrastive objective and text encoder. Its resolution is chosen so that its encoder FLOPs match our 1024-pixel merged model. This is roughly a 512-pixel input with no token reduction, with the exact size fixed by profiling. It embodies "just downsample," and it is the baseline most likely to embarrass us on findings that are large and diffuse. Beating it means higher 1%-label ChestX-ray14 AUC and SIIM pneumothorax Dice at equal measured FLOPs and wall-clock.
The strongest efficiency competitor is text-agnostic merging trained in the loop at 1024 pixels. We run two versions under our objective, each tuned with the same hyperparameter budget. The first uses ToMe's key-similarity bipartite matching (Token Merging), which doubles ViT-L/H throughput for a 0.2–0.3% ImageNet drop. The second uses DTMFormer's attention-guided merging with reconstruction (DTMFormer), which cut SETR GFLOPs from 53.2 to 9.8 while reaching 92.53% Dice on ACDC. Both use the same merge schedule and token counts per layer as ours, so the only difference is what decides which tokens merge.
The strongest text-guided competitor is TRIPS adapted to radiology (TRIPS). It uses the clinical text encoder's summary token over the whole report as the query, with its dropped-token fusion. TRIPS reported 76.23 VQA test-dev versus 74.54 for ALBEF, at 20.89G versus 36.63G FLOPs, on general-domain data. For image-only evaluation it cannot receive a report, so we run it two ways: with a report at test time as an oracle, and with a scorer distilled from its summary-token maps. The distilled version isolates the value of entity-level, negation-aware supervision.
The strongest task-accuracy references are published medical vision-language models. On SIIM pneumothorax with 1% labels, ASG reports 71.9% Dice and ECAMP reports 67.3% Dice. On ChestX-ray14 with 1% labels, the file gives only relative figures: ASG beats MGCA by 0.6 AUC, and ECAMP reports a 2.5% margin over competitors. We therefore re-run ASG's released protocol ourselves rather than quote an absolute number. For RSNA pneumonia, ECAMP reports an 8.3% mAP lead over other ViT methods but no absolute value; no reported absolute RSNA mAP found on Lacuna. For CheXpert and COVIDx, no reported result found on Lacuna.
Three ablations isolate the key ingredients. The first uses all entities, including negated ones, as queries. The second replaces per-query ranks with a shared raw-cosine threshold. The third removes protection entirely, which reduces our model to the ToMe baseline.
Why not just…?
Why not just downsample to a resolution where full attention is cheap? This is a real threat for large findings such as cardiomegaly or consolidation, and we expect to tie there. The case against it rests on focal findings. ECAMP adds a super-resolution branch precisely because low-resolution features miss small lesions. TRIPS showed that compute saved on tokens can be spent on higher resolution for accuracy gains, though only on web data. Our stratified evaluation by lesion size settles this. If the downsampled baseline matches us on small pneumothoraces and nodules, the resolution argument fails.
Why not just fine-tune ToMe or DTMFormer merging into the model, with no text involved? This is the most serious alternative, because homogeneous tissue is visually redundant and similarity-based merging may already leave focal findings alone. The counter-argument is mechanistic. A thin pleural line or a small nodule has keys similar to surrounding lung, so similarity matching tends to absorb it. Bolya et al. flag dense prediction as ToMe's open weakness for this reason. We test it directly: on SIIM masks and RSNA boxes, never seen in training, we measure what fraction of lesion tokens survive unmerged under each method. If ToMe keeps lesions intact as often as our scorer does, and matches our downstream numbers, we will report that report supervision adds nothing.
Why not just use TRIPS-style text-query pruning with a clinical text encoder? Two reasons. First, it needs a report at inference, which image-only diagnosis lacks. Second, a summary embedding of a report that is mostly negations points toward findings that are not there. The distilled-TRIPS baseline and the include-negated-entities ablation measure the second effect separately from the first.
Why not just adapt a general vision-language model with lightweight fine-tuning? Tuning LayerNorm alone (Tuning LayerNorm in Attention) or LoRA can adapt a model cheaply. The authors note, however, that it assumes an already well-aligned base model. A Comparative Study of PEFT Methods finds that domain pre-training predicts downstream success far better than the choice of adaptation method. These methods also leave the quadratic cost at 1024 pixels untouched. We include one such run for reference only.
Proposed Experiments
Setup. All models pre-train on MIMIC-CXR with the same text encoder, number of steps and hyperparameter search budget. Encoder FLOPs are matched and reported alongside measured latency.
Downstream evaluation. We evaluate on ChestX-ray14 with 1%, 10% and 100% labels, measuring AUC. CheXpert and COVIDx test classification transfer. SIIM-ACR pneumothorax segmentation is measured with Dice, using merge-history reconstruction. RSNA pneumonia detection is measured with mAP. OpenI tests out-of-distribution classification; the file's only OpenI figure, BERTHop's 98.12% average AUC, comes from a curated 3,684-pair subset, so comparisons must use that subset. We also run zero-shot finding classification and image-report retrieval on the MIMIC-CXR test split.
First decisive check: lesion-size stratification. We split SIIM and RSNA cases by lesion area. The claim predicts that our gains over both the downsampled and ToMe baselines grow as lesions get smaller. A flat profile falsifies it.
Second check: protection fidelity. We measure the overlap between the image-only scorer's protected tokens and ground-truth lesion regions, none of which were used in training. Protection that does not track lesions better than ToMe's leftover unmerged tokens undercuts the mechanism even if accuracy improves.
Third check: the two report-handling ablations. Including negated entities should hurt most on reports with many negations. A shared cosine threshold should skew the budget toward a few findings, which we measure as the entropy of budget share across finding types.
Fourth check: geometry. For each model we compute image-text gap distance and V-measure of image embeddings clustered by CheXpert-style labels. We then add ModGap's true-pair alignment and centroid-uniformity losses from Closing the Modality Gap to the best model. This tests whether gap closing improves diagnosis-level grouping under radiology's label imbalance, or instead hurts fine-grained multi-label discrimination, as the paper's own caveat and Mind the Gap's "closing isn't always better" result would predict.
Stress test: adversarial robustness. Zhang et al. show that token compression can concentrate adversarial distortion. There, LLaVA's robust accuracy under a 16-token budget fell to 42.7% from the 49.7% seen under a prior attack. We therefore attack the learned scorer with a salience-inflating perturbation and report how much the protected set and downstream AUC shift, compared with the text-agnostic merging baselines.
References
- TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection
- Token Merging: Your ViT But Faster
- DTMFormer: Dynamic Token Merging for Boosting Transformer-Based Medical Image Segmentation
- Anatomical Structure-Guided Medical Vision-Language Pre-training
- ECAMP: Entity-centered Context-aware Medical Vision Language Pre-training
- BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis
- A Vision-Language Foundation Model for Zero-shot Clinical Collaboration and Automated Concept Discovery in Dermatology
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Closing the Modality Gap Aligns Group-Wise Semantics
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning
- A Comparative Study of PEFT Methods for Python Code Generation
- On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression