Models that learn from radiology reports can now point at findings in chest X-rays without anyone drawing a box. They work by matching words in the report to patches of the image. Their weak spot is that "where" is learned only from statistics: which patches tend to co-occur with which words. At one hospital those statistics include shortcuts such as tubes, portable-film markers and patient posture, and these do not transfer to the next hospital. Most chest studies also contain a free second observation of the same anatomy: the lateral (side) view, taken seconds after the frontal one. Both views are horizontal X-ray projections of the same standing patient, so a finding at the level of the right lung base sits at roughly the same height in both images. Height is the one coordinate the two views share, and it is fixed by physics, not by any particular hospital's habits.
We propose to use that shared height as a training-only consistency check on report-driven grounding. When the model grounds a phrase like "left basilar opacity" on the frontal view, the vertical profile of its attention map must agree, after a per-study one-dimensional registration, with the profile of the same phrase grounded on the lateral view. Nobody annotates anything to obtain this signal. At test time the model needs only one frontal image, so it can still be deployed on portable single-view exams. The claim to test is specific: a constraint grounded in projection geometry should make grounding more correct, and more correct in a way that survives moving to new hospitals, than adding the lateral view through learned attention alone.
Core Claim
Training on a single center, here MIMIC-CXR from one Boston hospital, teaches grounding that partly tracks institution-specific correlates. A cross-view height-consistency loss penalizes any grounding map whose vertical position cannot be reproduced in the orthogonal projection. Device-driven and posture-driven shortcuts often satisfy the report statistics but not the geometry, so this loss should shrink the gap between internal grounding quality (MS-CXR, same hospital) and external grounding quality (Stanford's CheXlocalize, NIH ChestX-ray8 boxes, Vietnam's VinDr-CXR). The size of the shrinkage is the central measurement. Any absolute gain on internal benchmarks is secondary.
Positioning Against Prior Work
The closest prior work already combines two of our four ingredients, so the novelty has to be stated carefully.
LIMITR attends between every report word and every patch of both frontal and lateral images, and learns to weight views per phrase. MVCM adds a contrastive loss pulling two views of a study together, plus token-level text–patch alignment. EVOKE fuses multiple views to generate reports. All three treat the lateral view as more input, coupled to the frontal view by unconstrained learned attention. None imposes projective geometry, and all were trained and tested inside MIMIC-CXR. These papers supply our backbone and are our main competitors. We keep their local alignment and replace "lateral as input" with "lateral as a geometric referee".
A second group motivates the work by showing that strong report-supervised grounding is currently single-view. RadPRISM grounds 19 concepts on frontal images using a private Munich archive, and its authors name single-site pretraining as a limitation. LGDEA aligns LLM-extracted evidence phrases under limited pairing. ELVIS uses frontal views only. ELVIS also constrains our design. It showed that naive local cross-modal losses distort within-image similarity structure, for example bilateral findings. Our loss therefore acts only on height marginals and leaves left–right structure untouched.
CheXmask documents systematically worse anatomical segmentation on portable AP views than on PA views across five public centers. This is direct evidence of the protocol bias we stratify by. SpurCon shows that a pneumothorax classifier leans on chest tubes, and fixes this with patient-ID-weighted contrastive learning on global embeddings. It does no localization and needs longitudinal data. We pursue the same goal of ignoring artifacts with a different lever, geometry, applied to grounding maps. Counterfactual Contrastive Learning shares our logic of testing whether a grounding is causal by intervening on it, but on natural images. Our "intervention" is a second physical projection rather than a synthetic edit.
Mechanism
Take a word–patch alignment model of the LIMITR type. For a report phrase $t$, let $A^F_t(y,x)$ be its normalized grounding map on the frontal image and $A^L_t(y,z)$ the map on the lateral image. Here $y$ is height, $x$ is left–right, and $z$ is front–back. Define the height profiles $p^F_t(y)=\sum_x A^F_t(y,x)$ and $p^L_t(y)=\sum_z A^L_t(y,z)$.
The two films differ slightly in magnification and vertical framing, so we estimate a per-study scale-and-shift $\phi$ by one-dimensional registration of the two images' row-wise feature profiles. This is closed-form or a tiny grid search. Phrase grounding plays no part in estimating $\phi$. The added loss is
$$\mathcal{L}_{\text{geo}}=\sum_{t}w_t\, W_1\!\left(p^F_t,\; p^L_t\circ\phi\right),$$
where $W_1$ is the one-dimensional Wasserstein distance, which is cheap and tolerant of small misalignments. The weight $w_t$ is the phrase-importance weight LIMITR's aggregation already produces, so normal boilerplate contributes little.
The loss could be satisfied trivially by flattening both maps. Two things keep the maps sharp: the standard image–text contrastive loss stays on, and a floor on profile entropy is enforced. Phrases asserting laterality only ("left", "right") carry no height information and are excluded from the loss by a simple rule. The lateral branch and $\mathcal{L}_{\text{geo}}$ are used only during training.
Research Program
The first stage builds and trains the model on MIMIC-CXR studies that contain both a frontal and a lateral image. Phrases come from the same report-parsing route the baselines use, so that differences come from the loss and not the parser. As a check on the "pairing bottleneck", we also train a variant in which $\mathcal{L}_{\text{geo}}$ applies to two-view studies whose reports are withheld. In that case agreement between the model's own frontal and lateral groundings of its predicted findings is the only signal.
The second stage measures geometry directly. On a small set of MIMIC studies, a radiologist marks the vertical extent of findings on paired views. This is a few hundred studies, and drawing one line per finding is much cheaper than drawing boxes. The marks tell us how accurate the shared-height assumption actually is, and for which findings it breaks, such as posterior costophrenic recesses that sit lower on the lateral view. If the assumption is badly violated for some classes, we learn a per-class offset rather than abandon the idea.
The third stage is the cross-center evaluation described below, stratified by AP versus PA, by finding type, and by the presence of support devices.
Baselines
The do-nothing reference is a frontal-only local alignment model, essentially the single-view variant of the same backbone, trained on the same MIMIC frontal images at the same resolution. Every gain is measured against it.
The strongest existing weakly supervised method that also uses the lateral view is LIMITR. It reports an average MS-CXR phrase-grounding contrast-to-noise ratio (CNR) of 1.206, compared with 1.142 for BioViL. It is the right competitor because it already uses lateral views and word–patch attention, so it isolates whether geometry adds anything over learned cross-view attention. We retrain it ourselves on the identical study list, resolution, and epoch and tuning budget, with the same number of hyperparameter trials per method. We report our retrained number next to the published 1.206.
MVCM is a second multi-view competitor whose view-consistency is global rather than geometric. It is retrained under the same budget. It never reported phrase-grounding numbers, so we generate them.
ELVIS, a single-view model with a locality-preserving loss, reports an average MS-CXR CNR of 1.117. It tests whether a within-image structural prior already achieves what our cross-view prior does. RadPRISM reports a cardiomegaly pointing-game hit rate of 0.983 on CheXlocalize. Its training data is private, so it serves as a published reference rather than a retrained baseline. Fully box-supervised MedGround-R1 reports MS-CXR mIoU 79.02 and ChestX-ray8 mIoU 53.12. It is listed as an upper reference from a different supervision regime, not a target.
For VinDr-CXR, no reported grounding result was found on Lacuna for any of these methods, so all VinDr numbers will be new.
The key ablation keeps the lateral branch and every other loss but sets $\mathcal{L}_{\text{geo}}$ to zero. Beating LIMITR counts only if our method also beats this ablation. "Beating" means a larger reduction of the internal-to-external grounding gap, averaged over the three external centers, with study-level bootstrap intervals that exclude zero, and no loss of internal MS-CXR CNR.
Why not just…?
Why not just feed the lateral view in with cross-attention, as LIMITR does? This is the most serious alternative. Learned attention can, in principle, discover the height correspondence by itself. The file gives no evidence about whether it does, and LIMITR was never tested outside MIMIC-trained conditions on external grounding. The ablation with the lateral branch on and $\mathcal{L}_{\text{geo}}$ off settles this directly. If it closes the cross-center gap as much as the full model, the geometric loss is unnecessary, and we will report that.
Why not just train a single-view grounder on more data or at higher resolution? RadPRISM shows that a single-view model with heavy concept supervision grounds well. Scale may absorb much of our effect internally. Scale on one center does not, however, remove that center's correlates, and that is what the external gap measures. We will match the frontal-only baseline's compute to ours, spending the lateral-branch budget on extra frontal epochs and resolution. A cross-center advantage that survives compute matching would show the effect is not simply more training.
Why not just debias with metadata or augmentation, such as SpurCon-style patient-ID contrast or anatomical priors from CheXmask masks? SpurCon addresses global classification and needs repeat visits. CheXmask gives frontal lung and heart masks that could keep attention inside the lungs. That prior would suppress off-lung device attention but cannot say where within the lung a finding lies. We include a CheXmask-masked-attention variant as an extra baseline. If it matches us on external grounding, the geometric story reduces to "keep attention inside anatomy", and we would say so.
Proposed Experiments and Decisive Checks
All models train on MIMIC-CXR only. Internal evaluation uses MS-CXR (CNR and box mIoU from thresholded maps). External evaluation uses CheXlocalize (pointing game and mask IoU), ChestX-ray8's 984 boxed images, and VinDr-CXR's local boxes for overlapping finding classes. The primary metric is the internal-minus-external drop, averaged over the three external centers.
We stratify every result by AP versus PA. CheXmask's finding predicts that baselines degrade most on AP views, and a geometry-trained model should degrade less there even though it sees only a frontal image at test time. On CheXpert-derived images with tubes, we also measure the fraction of grounding mass falling on device regions identified by a few-shot prototype labeling in SpurCon's Pick-and-Predict style.
The decisive check is a mismatched-lateral control. We retrain with each frontal paired to a lateral from a different patient, registered the same way. If this control yields the same external gains, the loss is acting as a generic regularizer and the geometric hypothesis is false. A second control applies random vertical shifts to the true lateral; gains should shrink as the shift grows.
Finally, the per-finding analysis should show the largest gains where height is diagnostic and ambiguous, such as basilar atelectasis, effusion and retrocardiac opacity. Gains should be smallest for cardiomegaly, whose position is fixed. If that ordering fails, the mechanism we propose is not the one producing the results.
References
- LIMITR: Leveraging Local Information for Medical Image-Text Representation
- MVCM: Enhancing Multi-View and Cross-Modality Alignment for Medical Visual Question Answering and Medical Image-Text Retrieval
- MCL: Multi-view Enhanced Contrastive Learning for Chest X-ray Report Generation
- RadPRISM: Schema-stratified radiology-report supervision for concept-disentangled image representations and visual grounding
- LLM-Guided Diagnostic Evidence Alignment for Medical Vision–Language Pretraining under Limited Pairing
- ELVIS: Empowering Locality of Vision Language Pre-training with Intra-modal Similarity
- CheXmask: a large-scale dataset of anatomical segmentation masks for multi-center chest x-ray images
- SpurCon: Weighted Supervised Contrastive Learning for Mitigating Spurious Cues in Medical Imaging
- Counterfactual Contrastive Learning for Weakly-Supervised Vision-Language Grounding
- MedGround-R1: Advancing Medical Image Grounding via Spatial-Semantic Rewarded Group Relative Policy Optimization