CORAL8: Concurrent Object Regression for Area Localization in Medical Image Panels

Sam Maksoud, Arnold Wiliem, Kun Zhao, Teng Zhang, Lin Wu, Brian C. Lovell

TL;DR

This paper introduces CORAL8, a novel deep learning architecture for generating medical reports from multi-image panels, specifically for Renal Direct Immunofluorescence (RDIF) assays. It addresses the challenge of integrating information from multiple images and clinical notes to produce accurate and contextually relevant reports, outperforming existing methods.

What Problem Does This Paper Address?

Generating comprehensive medical reports, especially from complex medical imaging like Whole Slide Images (WSIs), is a time-consuming and expertise-intensive task for pathologists. The RDIF assay, for instance, requires analyzing eight different WSIs per patient to identify specific patterns. Existing image captioning models, while promising for general image descriptions, struggle with the nuanced interpretation required in medical domains. Key challenges include:

  • Contextual Interpretation: Medical image interpretation often relies on clinical context (e.g., patient history, other lab results) which can be crucial for differentiating morphologically similar pathologies.
  • Multi-Image Correlation: Effectively integrating information from multiple, temporally independent images within a panel is difficult. Standard sequential models might not capture the necessary relationships.
  • Ordered Set to Sequence Problem: The RDIF assay involves an ordered set of images (each representing a specific antibody stain), and the model needs to process this ordered set to generate a sequential report.

Methods

CORAL8 is a multi-modal generative recurrent neural network (RNN) architecture designed to tackle these challenges. Its key components include:

  • Image Encoder: This module processes an ordered set of N images. It extracts local image features (e.g., from VGG-16) and a global image set feature.
  • Prior Encoder: This component integrates clinical notes and previous sentence features to create a context vector. This context vector acts as a "prior" to guide the report generation, ensuring it aligns with the patient's clinical background.
  • Sentence Generator: An RNN that generates the report word by word. It uses an attention mechanism to dynamically sample image data concurrently across the RDIF panel. This attention mechanism is crucial for allowing the model to "look" at relevant parts of all images at each step of generation.
  • Attention Regularization: To prevent the attention mechanism from assigning uniform weights (effectively ignoring it), the authors introduce two novel regularization terms:
  • Salient Alpha (SAL): Encourages the network to be more selective by increasing the distance between the maximum and average attention weights.
  • Time Distributed Variance (TDVAR): Promotes variability in attention weights across different words, forcing the model to attend to different image features for each generated word.
  • These, along with a standard regularization from Xu et al. [17], are combined into a single Calpha loss term.

Datasets

The paper introduces a novel dataset for the RDIF assay.

  • RDIF Dataset: Assembled from routine clinical samples in collaboration with Sullivan Nicolaides Pathology. It comprises 144 patient samples, split into 99 training, 15 validation, and 30 test sets. Each sample includes:
  • Eight Whole Slide Images (WSIs) from renal biopsy sections stained with different fluorescein isothiocyanate (FITC) conjugated antibodies (IgG, IgA, IgM, Kappa, Lambda, C1q, C3, or Fibrinogen).
  • Clinical notes from the requesting physician.
  • The corresponding medical report.

Key Insights and Ideas

  • Clinical Context is King: The study strongly emphasizes that incorporating clinical notes as a "prior" significantly improves report generation accuracy, especially in differentiating subtle pathologies.
  • Attention is Not Enough (Without Guidance): Simply using an attention mechanism isn't sufficient. Regularizing attention weights is critical to ensure the model effectively utilizes information from all relevant image regions and modalities.
  • Ordered Set to Sequence: The problem of generating a report from an ordered set of images (like the RDIF panel) requires specialized handling, which CORAL8 addresses by processing the ordered image set and then generating a sequential report.

Key Results

  • CORAL8 demonstrates significant improvements over existing methods on the novel RDIF dataset, as evaluated by standard metrics like BLEU, ROUGE, and METEOR.
  • Ablation studies show that removing any of the proposed components (clinical notes, SAL, TDVAR) leads to decreased performance, highlighting the contribution of each element.
  • Qualitative examples (Table 2) illustrate how the model, with clinical notes, can correctly infer diagnoses (e.g., IgA nephropathy) that models without this context might miss or misinterpret.

Limitations and Future Work

  • Incorrect Impression Example: The paper acknowledges an instance where CORAL8 generated an incorrect impression (Example 3 in Table 2), suggesting that even with improvements, semantic understanding of complex pathologies remains a challenge. However, it notes that the underlying pathophysiology for both the ground truth and the predicted condition shared commonalities (hypertension), indicating the model captures some semantic relationships.
  • Future Work: The authors suggest further exploration into how clinical notes improve results and how to better model pathologies with shared clinical features. The focus on improving the semantic understanding of complex medical conditions is a clear direction.

Summary updated

Abstract

This work tackles the problem of generating a medical report for multi-image panels. We apply our solution to the Renal Direct Immunofluorescence (RDIF) assay which requires a pathologist to generate a report based on observations across eight different whole slide images (WSI) in concert with existing clinical features. To this end, we propose a novel attention-based multi-modal generative recurrent neural network (RNN) architecture capable of dynamically sampling image data concurrently across the RDIF panel. The proposed methodology incorporates text from the clinical notes of the requesting physician to regulate the output of the network to align with the overall clinical context. In addition, we found the importance of regularizing attention weights for the word generation processes. This is because the system can ignore the attention mechanism by assigning equal weights for all members. Thus, we propose two regularizations to encourage efficient use of the attention mechanism. Experiments on our novel collection of RDIF WSIs provided by Sullivan Nicolaides Pathology demonstrate that our framework offers significant improvements over existing methods.
BibTeX
@inproceedings{maksoud2019coral,
  author = {Sam Maksoud and Arnold Wiliem and Kun Zhao and Teng Zhang and Lin Wu and Brian C. Lovell},
  title = {{CORAL8: Concurrent Object Regression for Area Localization in Medical Image Panels}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  doi = {10.1007/978-3-030-32239-7\_48},
}
Figure