AnaXNet: Anatomy Aware Multi-label Finding Classification in Chest X-Ray

Nkechinyere N. Agu, Joy T. Wu, Hanqing Chao, Ismini Lourentzou, Arjun Sharma, Mehdi Moradi, Pingkun Yan, James A. Hendler

TL;DR

AnaXNet is a novel deep learning framework that mimics the way radiologists interpret chest X-rays by focusing on specific anatomical regions rather than the whole image. By combining object detection with Graph Convolutional Networks (GCNs), the model captures the complex inter-dependencies between anatomical structures and pathologies, outperforming state-of-the-art global classification models.

What Problem Does This Paper Address?

When a radiologist looks at a chest X-ray (CXR), they don't just see a grid of pixels; they perform a structured search across specific anatomical zones—like the lung bases, the heart silhouette, or the hilar regions. Most existing AI models, however, treat the entire X-ray as a single global input. This "black box" approach often ignores crucial anatomical context and fails to capture how a finding in one region (e.g., fluid in the pleural space) might relate to findings in another (e.g., an enlarged heart).

The authors identify two main gaps:

  1. Lack of Anatomical Awareness: Standard models don't explicitly "know" where the lungs or heart are.
  2. Ignoring Inter-dependencies: Pathologies are often correlated. Existing models struggle to leverage these relationships to improve diagnostic accuracy and interpretability.

Methods

The authors propose AnaXNet (Anatomy-aware X-ray Network), which consists of three primary stages:

  1. The Detection Module: Using a Faster R-CNN (with a ResNet-50 backbone), the model identifies and extracts features from 18 specific anatomical regions of interest (ROIs), such as the right upper lung zone, the mediastinum, and the cardiac silhouette.
  2. The Anatomy GCN: These 18 regions are treated as nodes in a graph. To determine how these nodes should "talk" to each other, the authors created an adjacency matrix based on the Jaccard similarity of disease co-occurrence. If two regions often share the same pathologies in the training data, the connection between them is stronger. The GCN then updates the features of each region by aggregating information from its neighbors.
  3. Cross-Anatomy Attention: To make a final decision, the model uses a non-local operation (similar to self-attention) to weight the importance of different anatomical features before passing them through a fully connected layer for multi-label classification.

This approach is "anatomy-aware" because the model's internal reasoning is explicitly tied to the physical structure of the human chest.

Datasets

The study utilizes the Chest ImaGenome dataset, which is a large-scale structured dataset derived from the MIMIC-CXR database.

  • Scale: 217,417 frontal chest X-ray images.
  • Annotations: It includes 18 anatomical regions and 9 pathology labels (e.g., Lung Opacity, Pleural Effusion, Pneumonia).
  • Uniqueness: Unlike standard datasets that only provide global image labels, this dataset links findings to specific bounding boxes, allowing for the training of the detection and graph modules.

Key Insights and Ideas

  • Anatomy as a Graph: The most significant "aha moment" is treating human anatomy as a graph. Because the physical relationship between organs is constant, a graph is a much more natural representation for medical images than a simple flat vector of features.
  • Localization Improves Classification: The paper demonstrates that by forcing the model to "look" at the right places (via the detection module), the overall classification accuracy improves.
  • Context Matters: A global model might correctly identify "Enlarged Heart" but actually be looking at the lungs to make that decision (as shown via Grad-CAM analysis). AnaXNet ensures the model is right for the right reasons by focusing its attention on the cardiac silhouette.

Key Results

AnaXNet was compared against several baselines, including a standard global DenseNet (GlobalView) and CheXGCN (a model that uses graphs for labels but not anatomy).

  • Superior Accuracy: AnaXNet achieved an average AUC of 0.93, outperforming the GlobalView baseline (0.89) and CheXGCN (0.90).
  • Specific Pathologies: The model showed particular strength in detecting Enlarged Cardiac Silhouette (0.99 AUC) and Pleural Effusion (0.96 AUC).
  • Interpretability: Qualitative results showed that AnaXNet provides accurate localization. Even when global models correctly predicted a disease, they often failed to localize it correctly, whereas AnaXNet's attention remained anchored to the relevant anatomy.

Limitations and Future Work

  • Detection Dependency: The model's performance is heavily dependent on the initial Faster R-CNN detection. If the detector fails to find a specific anatomical region, the GCN receives a "zero vector," which can hamper the reasoning process.
  • Label Noise: The ground truth labels were extracted from radiology reports using Natural Language Processing (NLP). While the authors validated a subset with clinicians, NLP extraction is rarely 100% accurate.
  • Future Directions: The authors suggest expanding the model to include more pathologies and integrating it into automated report generation systems, where the "structured" nature of the model's output could help write more coherent, anatomically-accurate medical reports.

Summary updated

Abstract

Radiologists usually observe anatomical regions of chest X-ray images as well as the overall image before making a decision. However, most existing deep learning models only look at the entire X-ray image for classification, failing to utilize important anatomical information. In this paper, we propose a novel multi-label chest X-ray classification model that accurately classifies the image finding and also localizes the findings to their correct anatomical regions. Specifically, our model consists of two modules, the detection module and the anatomical dependency module. The latter utilizes graph convolutional networks, which enable our model to learn not only the label dependency but also the relationship between the anatomical regions in the chest X-ray. We further utilize a method to efficiently create an adjacency matrix for the anatomical regions using the correlation of the label across the different regions. Detailed experiments and analysis of our results show the effectiveness of our method when compared to the current state-of-the-art multi-label chest X-ray image classification methods while also providing accurate location information.
BibTeX
@inproceedings{agu2021anaxnet,
  author = {Nkechinyere N. Agu and Joy T. Wu and Hanqing Chao and Ismini Lourentzou and Arjun Sharma and Mehdi Moradi and Pingkun Yan and James A. Hendler},
  title = {{AnaXNet: Anatomy Aware Multi-label Finding Classification in Chest X-Ray}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=CBJqNs1LUQ},
}

Loading related content...

Figure