Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rays

Ke Yu, Shantanu Ghosh, Zhexiong Liu, Christopher Deible, Kayhan Batmanghelich

Abstract

Deep learning models for medical image analysis typically require massive amounts of expert-annotated data, such as bounding boxes, which are prohibitively expensive to collect. Weakly-supervised learning (WSL) offers an alternative by using labels extracted from radiology reports via Natural Language Processing (NLP). However, these methods often ignore anatomical context and suffer from label noise. This paper introduces AGXNet (Anatomy-Guided chest X-ray Network), a novel framework that leverages anatomical context and Positive Unlabeled (PU) learning to improve abnormality localization. By mimicking the systematic workflow of radiologists, AGXNet outperforms traditional supervised and weakly-supervised baselines on large-scale datasets like MIMIC-CXR and NIH Chest X-ray.

Introduction

The primary challenge in scaling medical AI is the "annotation bottleneck." While radiology reports provide a wealth of information, translating them into training signals for computer vision models is difficult. Current WSL methods face two major hurdles. First, they often ignore the spatial relationship between diseases and anatomy (e.g., pneumonia occurring in the lower lung). Second, they suffer from "reporting bias": if a radiologist does not mention an abnormality, it may still be present but considered secondary, leading to sparse and noisy labels. AGXNet addresses these issues by integrating anatomical guidance directly into the neural architecture and employing robust statistical methods to handle label uncertainty.

Methodology

Dual-Network Architecture

AGXNet utilizes a cascaded architecture consisting of two specialized encoders:

  1. Anatomy Network ($g_a$): Trained to identify anatomical abnormalities across 46 distinct landmarks.
  2. Observation Network ($g_o$): Focused on identifying specific pathological observations (diseases).

Anatomy-Guided Attention (AGA)

The core innovation of AGXNet is the Anatomy-Guided Attention (AGA) module. This module generates Class Activation Maps (CAMs) from the anatomy network to identify relevant spatial regions. These maps are then fed into the observation network as a residual attention map. This mechanism forces the disease classifier to focus on the correct anatomical neighborhood, creating a gradient feedback loop where the two networks mutually reinforce each other's features.

Structured Knowledge and PU Learning

To provide the necessary ground truth for training, the model uses the RadGraph dataset to parse reports into an adjacency matrix that explicitly links observations to anatomical locations. To combat the "silence implies nothing" fallacy in reports, the authors employ Positive Unlabeled (PU) Learning. Using a "Best Bin Estimation" technique, the model estimates the proportion of hidden positives in the unlabeled data and iteratively refines its labels through self-training.

Experimental Results

Datasets and Evaluation

The model was primarily trained and evaluated on MIMIC-CXR, featuring over 220,000 frontal chest X-rays. Localization accuracy was validated using Chest ImaGenome bounding boxes. To test the model's robustness, the authors evaluated its transferability on the NIH Chest X-ray dataset, fine-tuning the MIMIC-pre-trained encoders on new data.

Performance Highlights

  • Superior Localization: AGXNet achieved state-of-the-art (SOTA) localization results on MIMIC-CXR, outperforming both standard weakly-supervised baselines and even a fully supervised RetinaNet for conditions like pneumonia and pneumothorax at various Intersection over Union (IoU) thresholds.
  • Robust Generalization: When transferred to the NIH dataset, AGXNet maintained high performance with a mean AUROC of 0.84, proving that anatomy-aware features are robust across different clinical environments.
  • Ablation Insights: Removing the AGA module resulted in a significant drop in localization accuracy, confirming that anatomical guidance is the primary driver of the model's spatial intelligence.

Discussion and Limitations

AGXNet demonstrates that incorporating clinical domain knowledge—specifically the relationship between anatomy and pathology—can significantly boost the performance of weakly-supervised models. The use of PU learning specifically helped with conditions like pneumonia, which have high reporting variability, though it was less critical for conditions like pneumothorax that are almost always reported when present.

However, the model remains dependent on the quality of the initial NLP extraction from RadGraph; errors in report parsing can propagate through the system. Additionally, while the model does not require text at inference time, future iterations could benefit from incorporating clinical metadata to further refine predictions. Currently, the evaluation is focused on common pathologies, and further research is needed to determine how well this framework scales to rarer or more complex medical findings.

Conclusion

AGXNet represents a significant step toward more interpretable and clinically grounded AI for radiology. By bridging the gap between raw image data and the structured anatomical reasoning used by human experts, the framework provides a scalable path for developing high-performance diagnostic tools without the need for exhaustive manual annotation.

Summary updated

BibTeX
@inproceedings{yu2022anatomy,
  author = {Ke Yu and Shantanu Ghosh and Zhexiong Liu and Christopher Deible and Kayhan Batmanghelich},
  title = {{Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rays}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  doi = {10.1007/978-3-031-16443-9\_63},
}
Figure