Abnormality-Driven Representation Learning for Radiology Imaging

Marta Ligero, Tim Lenz, Georg Wölflein, Omar S. M. El Nahhas, Daniel Truhn, Jakob Nikolas KatherarXiv 2024 · DOI · arXiv

Introduction

Radiology and pathology have historically diverged in their approach to AI. While pathology leverages task-agnostic foundation models trained on 2D patches with attention mechanisms, radiology has largely relied on resource-intensive, end-to-end 3D networks tailored for specific tasks. These 3D models require massive annotated datasets and significant computational power.

To bridge this gap, researchers introduced CLEAR (Contrastive Learning-based Embeddings for Attention-based Radiology), a framework designed to learn general "abnormality" representations across the whole body using efficient 2D slices. This approach brings the efficiency of digital pathology workflows to the complex world of radiology imaging.

The CLEAR Framework and LECL

The core of CLEAR is the Lesion-Enhanced Contrastive Learning (LECL) strategy. Unlike standard contrastive learning, which simply matches different views of the same image, LECL is specifically "lesion-aware." During pre-training, it prioritizes crops centered on known abnormalities and employs a specialized loss function to distinguish abnormal slice embeddings from the rest of the training set.

The CLEAR framework (B) compared to traditional 3D approaches (A), alongside the LECL contrastive learning mechanism.

The architecture moves beyond standard Vision Transformers (ViTs), exploring more efficient alternatives like Vision State Space Models (VMamba) and Gated CNNs (MambaOut). To process 3D volumes, CLEAR extracts 2D embeddings for each slice and aggregates them using Attention-based Multiple Instance Learning (ABMIL), which learns to weigh the clinical relevance of each slice.

Experimental Setup and Datasets

The model was pre-trained on DeepLesion, a dataset containing over 870,000 axial CT slices. Evaluation spanned three distinct tasks to test both internal and external generalization:

  1. Internal Validation: Lesion classification on DeepLesion across various body regions (Abdomen, Mediastinum, Pelvis, etc.).
  2. External Validation (RadChest): Lung disease detection (e.g., Emphysema, Fibrosis).
  3. External Validation (NSCLC-Radiomics): Patient cancer staging.

Key Insights and Results

The study yielded several critical findings that challenge the necessity of heavy 3D models:

  • 2D Efficiency: High-quality 2D embeddings paired with smart attention mechanisms can effectively capture spatial context, often outperforming heavy 3D convolutions while being significantly more data- and compute-efficient.
  • Architecture Matters: VMamba and MambaOut consistently outperformed standard ViTs, suggesting these architectures are better suited for the specific textures and structures found in medical imaging.
  • Superior Performance: On lesion classification, the MambaOut-based LECL model achieved a +0.8% AUC improvement over BiomedCLIP. In specialized areas like soft tissue, the gain was a significant +5.9% AUC.
  • Generalization: The model demonstrated strong performance on external datasets (RadChest and NSCLC) without requiring task-specific fine-tuning of the backbone.

Interpretability by Design

One of the primary advantages of the CLEAR framework is its inherent interpretability. By utilizing an attention-based aggregator, the model provides a clear signal regarding which slices were most influential in its final decision.

Attention distribution across CT slices, highlighting the model's focus on specific anatomical regions.

This transparency allows clinicians to verify the AI's reasoning, transforming the "black box" into a tool that aligns with clinical workflows. Doctors can see exactly which slices the AI used to make its decision, making the model's output more trustworthy and actionable.

Limitations and Future Directions

The researchers identified a "Goldilocks" effect with the lesion-weighting parameter ($\lambda$); excessive weighting can cause the model to ignore necessary healthy anatomical context. Future work will explore extending CLEAR to other modalities like MRI and PET, and integrating full 3D segmentation capabilities to further support clinical needs. Additionally, the authors suggest that further research into domain-specific architectures is key to unlocking the full potential of AI in radiology.

Summary updated

Abstract

To date, the most common approach for radiology deep learning pipelines is the use of end-to-end 3D networks based on models pre-trained on other tasks, followed by fine-tuning on the task at hand. In contrast, adjacent medical fields such as pathology, which focus on 2D images, have effectively adopted task-agnostic foundational models based on self-supervised learning (SSL), combined with weakly-supervised deep learning (DL). However, the field of radiology still lacks task-agnostic representation models due to the computational and data demands of 3D imaging and the anatomical complexity inherent to radiology scans. To address this gap, we propose CLEAR, a framework for radiology images that uses extracted embeddings from 2D slices along with attention-based aggregation for efficiently predicting clinical endpoints. As part of this framework, we introduce lesion-enhanced contrastive learning (LeCL), a novel approach to obtain visual representations driven by abnormalities in 2D axial slices across different locations of the CT scans. Specifically, we trained single-domain contrastive learning approaches using three different architectures: Vision Transformers, Vision State Space Models and Gated Convolutional Neural Networks. We evaluate our approach across three clinical tasks: tumor lesion location, lung disease detection, and patient staging, benchmarking against four state-of-the-art foundation models, including BiomedCLIP. Our findings demonstrate that CLEAR using representations learned through LeCL, outperforms existing foundation models, while being substantially more compute- and data-efficient.
BibTeX
@misc{ligero2024abnormality,
  author = {Marta Ligero and Tim Lenz and Georg Wölflein and Omar S. M. El Nahhas and Daniel Truhn and Jakob Nikolas Kather},
  title = {{Abnormality-Driven Representation Learning for Radiology Imaging}},
  year = {2024},
  eprint = {2411.16803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.16803},
}
Figure