Models that learn from chest X-rays paired with radiology reports can learn to point at findings without any hand-drawn boxes. They do this by matching each report sentence to the image patches it best explains, which is a text-driven version of class activation mapping. Class activation mapping is the trick of reading a classifier's internal feature maps to see which image regions drove its decision. Current systems score every sentence on its own. That throws away something the report already states: how its sentences relate. "Left lung is clear. There is a right basilar opacity" is a contrast, and the two sentences should point at different places. "The heart is enlarged. Pulmonary vascular congestion has increased" is an unstated cause-and-effect link, and the two findings sit in different regions that tend to appear together. "There is a left effusion. It has increased in size" is an elaboration, and both sentences should point at the same place. These relations are mostly implicit, meaning no connective word such as "because" or "however" marks them. Recognizing such unmarked links is the task of implicit discourse relation recognition.
We propose to use these relations as a free weak signal about how the attention maps of two sentences should overlap. The target failure is specific and well understood. When two findings almost always co-occur in training reports, a weakly supervised model learns to light up both regions for either sentence, so the map for "effusion" bleeds onto the heart. Discourse relations say which co-occurrences mean "same region" and which mean "different region, present together." Our claim is that a relation-conditioned overlap penalty on sentence maps reduces this cross-finding leakage. We further claim it does so in cases that simpler fixes miss: rewriting sentences to be self-contained, or reading laterality words.
Core Claim and Positioning
The closest work grounds report text sentence by sentence or concept by concept and ignores how sentences relate. Mammo-CLIP builds sentence-level spatial attribution maps for mammograms without boxes. It reaches 0.45 mAP at IoU 0.25 for mass localization on VinDr-Mammo, but each sentence is an independent query. RadPRISM goes further toward independence. It splits reports into 19 concept slots, each aligned in its own subspace, and gets strong pointing-game results on CheXlocalize, for example 0.983 on Cardiomegaly. Splitting the report this way is precisely what removes the cross-concept dependencies we want to exploit. JoImTeRNet aligns words and n-grams to patches. ELVIS keeps each modality's internal similarity structure so that bilateral findings are not pushed apart. Neither looks at how the clauses of a report relate to one another. ELVIS is the most relevant constraint on our design. It shows that naive locality objectives damage representations by assuming distant patches are unrelated. So our penalty must not force disjointness by default; it should apply only where a relation licenses it.
On the discourse side, Entity Enhancement for Implicit Discourse Relation Classification in the Biomedical Domain motivates the language component. Its finding is that a biomedical language model alone (BioBERT) underperforms on implicit relations, and that injecting entity triples raises BioDRB accuracy to 69.57%. It also shows that silver labels mined from explicit relations in PubMed are weak, at 41.95% accuracy. This warns us that the relation classifier will be noisy, so the vision objective must tolerate noise. That paper is text-only. We use discourse relations as supervision for a different modality rather than as an end task. Earlier work on cue-phrase stripping found that text with its connectives removed differs from naturally implicit text. This tells us not to trust a relation classifier trained only on stripped explicit pairs without checking it on natural pairs.
To our knowledge, nothing in the literature uses inter-sentence discourse relations to constrain weakly supervised spatial attribution. That is the specific new piece.
Research Program
The vision-language backbone is a dual encoder pre-trained on MIMIC-CXR-JPG frontal image-report pairs. It has a sentence-query attention head in the style of Mammo-FActOR. For sentence $i$, the head produces an attribution map $A_i$, a distribution over image patches that sums to one. Adjacent sentence pairs $(i,j)$ in a report receive a coarse relation $r_{ij}$ from a relation classifier. The classes are same-referent (restatement, elaboration, temporal change of one finding), distinct-but-linked (cause/result, contrast between regions), and none/other. The classifier also outputs a confidence $p_{ij}$. Let $o_{ij} = \sum_u \min(A_i(u), A_j(u))$ be the shared attention mass of the two maps, where $u$ ranges over patches. The added loss is
$$L_{\text{disc}} = \sum_{(i,j)} p_{ij}\,\big[\,\mathbb{1}[r_{ij}=\text{same}]\,\max(0, \tau_s - o_{ij}) + \mathbb{1}[r_{ij}=\text{distinct}]\,\max(0, o_{ij} - \tau_d)\,\big].$$
Here $\tau_s$ and $\tau_d$ are target overlap thresholds for same-referent and distinct pairs. The none/other pairs get no constraint, which follows from the ELVIS lesson. The loss is added to the usual global and local contrastive objectives with a single weight. We also test a variant that learns per-relation targets in place of fixed thresholds, to check that the hand-set values are not doing hidden work.
Domain-specific pre-training of the relation classifier is a supporting requirement, not the centerpiece. Radiology prose ("likely representing," "compatible with," "unchanged from prior") is a narrow register. General encoders handle it poorly, as the BlueBERT-over-BERT gain of 6.46% AUC in BERTHop suggests. The classifier is built in three steps. We start from a clinical encoder (BlueBERT or BioClinicalBERT). We continue pre-training it with a connective-prediction objective on explicit sentence pairs mined from MIMIC-CXR reports. We then fine-tune it on a small new annotated set of about 600 naturally implicit adjacent-sentence pairs from MIMIC-CXR, labeled by two annotators with agreement reported. As a sanity check of general discourse ability, the same classifier is evaluated on BioDRB against the reported 69.57%. Matching that number is not required. We report it so readers can judge how good the relation signal is.
Baselines
The naive reference point is the same dual encoder without $L_{\text{disc}}$, scoring each sentence independently. This is a chest X-ray version of Mammo-FActOR, trained on the same data. It answers whether relations add anything at all. The strongest published weakly supervised competitor with a number on our main benchmark is ELVIS. It reports an average contrast-to-noise ratio (CNR) of 1.117 on MS-CXR, where CLIP and ConVIRT hover near zero. We will retrain ELVIS on the same MIMIC-CXR-JPG split, backbone, and step count rather than copy its number, and report both. For context only, MedRPG reaches 69.86% accuracy at IoU > 0.5 on MS-CXR (Medical Phrase Grounding with Region-Phrase Context Contrastive Alignment). That result uses box supervision, so it is an upper reference rather than a competitor. RadPRISM's CheXlocalize hit rates come from training on a private 323,562-image archive. We cite them as context and do not claim a controlled comparison.
Four ablations isolate the key ingredient. A relation-free overlap loss applies one shared overlap target to every adjacent pair; it tests whether any adjacency regularizer helps. A shuffled-relation control permutes relation labels within reports while keeping the same label frequencies; it tests whether the content of the relation matters. An oracle-relation run uses human labels on the annotated subset, which bounds what better discourse recognition could buy. A general-encoder relation classifier, with no clinical pre-training, tests whether the domain language component matters. Every variant gets the same pre-training data, the same number of steps, and the same budget of 20 hyperparameter trials, all selected on a held-out MIMIC split. Three seeds are run for each. Beating the baseline means two things at once. First, MS-CXR CNR and CheXlocalize pointing-game hit rate must improve over retrained ELVIS and over the naive baseline, beyond seed variance. Second, the leakage metric defined below must drop, with no loss on single-finding images.
Why not just…?
Why not just feed the previous sentence into the text encoder as context? A text encoder that sees context can resolve "it" and may absorb some relational meaning. Nothing in it, however, tells the image side that two maps should differ. Leakage comes from label co-occurrence in the vision-language objective, not from misreading the text. We run this context-concatenation variant as a baseline. If it matches our leakage reduction, the explicit relation loss is unnecessary.
Why not just have a language model rewrite every sentence to be self-contained, then ground each one independently? This is the most serious threat. Rewriting "It has increased" as "The left effusion has increased" probably fixes most same-referent cases with no new objective. We therefore include LLM decontextualization as a baseline and also stack it under our method. Our claim survives only if the gain remains on distinct-but-linked pairs, where rewriting changes nothing because each sentence is already self-contained. If the gain disappears there, we will report that discourse adds nothing beyond coreference.
Why not just extract laterality and anatomy words and constrain maps with a lung atlas? Rules handle "left clear, right opacity" well. We split the evaluation by whether a pair contains explicit, distinguishing location words. The relation loss must help on pairs without them, for example cardiomegaly with congestion or effusion. Otherwise a rule-based location prior should be preferred. We also run the rule-based prior as its own baseline and in combination with our loss.
Why not just scale a general vision-language model? The evidence on file points against this in radiology. Mammo-CLIP's domain model with 10% of the data beats general CLIP with 100%, and general grounding models trail MedRPG on MS-CXR by 6.1%. We therefore do not spend compute re-testing scale. The general-encoder ablation covers the language half of this question.
Proposed Experiments and Decisive Checks
Evaluation uses MS-CXR (1,153 phrase-box triples; CNR and pointing accuracy), the 984 boxed images in ChestX-ray8, and CheXlocalize pixel segmentations (pointing-game hit rate and mean IoU of thresholded maps). Classification AUC on ChestX-ray14 and OpenI checks that global recognition does not regress.
The new diagnostic is cross-finding leakage. On CheXlocalize images with two or more segmented pathologies, it is the fraction of finding A's attribution mass that falls inside finding B's mask but outside A's own mask. We report it for frequently co-occurring pairs (cardiomegaly with edema or effusion, atelectasis with effusion) separately from rare pairs.
The decisive result has three parts. Leakage should fall clearly relative to the naive baseline, ELVIS, the decontextualization baseline, and the rule-based prior. That reduction should be concentrated in distinct-but-linked pairs that lack location words. The shuffled-relation control should erase it. A dose-response check strengthens the case: on the annotated subset, moving from predicted to oracle relations should reduce leakage further, by an amount related to the classifier's accuracy gap.
The note would count as refuted in three cases. Decontextualization plus location rules matches our method on every slice. Shuffled relations work as well as real ones. Leakage falls only because single-finding grounding gets worse. We would treat each of these as a negative result worth reporting. It would mean that what reports contribute to localization is captured by sentence content and reference resolution, not by how sentences connect.
References
- Mammo-CLIP: A Vision Language Foundation Model to Enhance Data Efficiency and Robustness in Mammography
- RadPRISM: Schema-stratified radiology-report supervision for concept-disentangled image representations and visual grounding
- Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment
- ELVIS: Empowering Locality of Vision Language Pre-training with Intra-modal Similarity
- Entity Enhancement for Implicit Discourse Relation Classification in the Biomedical Domain
- BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis
- Medical Phrase Grounding with Region-Phrase Context Contrastive Alignment