ACAT: Adversarial Counterfactual Attention for Classification and Detection in Medical Imaging

Alessandro Fontanella, Antreas Antoniou, Wenwen Li, Joanna M. Wardlaw, Grant Mair, Emanuele Trucco, Amos J. Storkey

TL;DR

This paper introduces ACAT, a novel framework that leverages adversarial counterfactuals to generate saliency maps for medical image analysis. ACAT significantly improves classification accuracy and robustness in medical imaging tasks by using these saliency maps to guide attention mechanisms at multiple network scales, outperforming existing methods.

What Problem Does This Paper Address?

Medical imaging tasks, particularly those involving subtle pathologies like early-stage lesions, often present a challenge for standard Convolutional Neural Networks (CNNs). In these scenarios, the informative regions (Regions of Interest or ROIs) constitute only a small fraction of the image, while the rest can be considered "noise." Traditional CNNs can struggle to generalize well in such low signal-to-noise conditions. While manual ROI annotations can help, they are expensive, time-consuming, and prone to inter-annotator variability. Existing attention mechanisms and saliency map generation methods have limitations, either by fusing information at a single network stage, requiring manual annotations, or producing less reliable explanations. This paper aims to address the need for an automated, robust, and accurate method for identifying informative regions in medical images to improve classification performance.

Methods

The core of the ACAT framework is its innovative use of adversarial counterfactuals to generate saliency maps. Instead of relying on manual annotations, ACAT first trains a baseline classifier and an autoencoder. The autoencoder is used to generate "counterfactual" images by subtly perturbing the latent representation of an input image. These counterfactuals are designed to flip the classifier's prediction for a specific class (e.g., changing an image from "lesion present" to "lesion absent"). The difference between the original and counterfactual images, when analyzed in a class-agnostic manner, highlights the most discriminative pixels, forming a saliency map.

These saliency maps are then fed into a dedicated "saliency branch" within the ACAT architecture. This branch uses attention modules at different scales (early, middle, and late layers) to extract salient features. Crucially, these multi-scale attention masks are fused using an "attention fusion layer" to create a comprehensive representation of the informative regions. This fused mask then modulates the features in the main image processing branch, allowing the network to focus on relevant areas and suppress noise. The authors also introduce a regularization term to the counterfactual generation process to ensure the generated images remain within the data manifold and are semantically meaningful.

Datasets

The paper uses two publicly available datasets for evaluation:

  • IST-3 (Third International Stroke Trial): A dataset of brain CT scans for stroke lesion detection.
  • MosMed: A dataset of lung CT scans for COVID-19 related findings.

The authors do not introduce new datasets but utilize existing ones for training and testing their framework.

Key Insights and Ideas

  • Adversarial Counterfactuals as Saliency Generators: The paper demonstrates that generating adversarial counterfactuals, which subtly alter an image to change its predicted class, is a powerful and automated way to create highly informative saliency maps for medical images.
  • Multi-Scale Attention Fusion: Fusing saliency-based attention masks from different network depths is critical for capturing information at various scales, especially in low signal-to-noise medical imaging scenarios.
  • Robustness through Saliency Modulation: ACAT's approach of using saliency maps to modulate image features significantly reduces the variance of pre-activations within the network, leading to increased robustness against input perturbations.
  • Class-Agnostic Saliency Generation: The method effectively generates saliency maps without requiring ground-truth class labels during the saliency generation phase, making it more practical for real-world applications.
  • Semantic Meaningfulness of Counterfactuals: The proposed regularization in counterfactual generation ensures that the generated images are not only effective for saliency mapping but also semantically plausible, unlike some simpler latent shift methods.

Key Results

  • Improved Classification Accuracy: ACAT achieved state-of-the-art results on both IST-3 and MosMed datasets.
  • On IST-3 (brain CT scans), accuracy increased from a baseline of 71.39% to 72.55%.
  • On MosMed (lung CT scans), accuracy increased from 67.71% to 70.84%.
  • Superior Saliency Map Localization: In a task of localizing lesions among 6 possible regions in brain CT scans, ACAT achieved a score of 65.05%, outperforming the best competing method (latent shift) at 61.29%.
  • Enhanced Robustness: ACAT significantly reduced the variance of pre-activations in network layers compared to the baseline, indicating improved robustness to input noise.
  • Competitive Performance with Limited Data: ACAT showed strong performance even with limited training data (50-300 scans), often outperforming other methods.
  • High IoU and Dice Scores: ACAT achieved the best IoU (0.5203) and Dice coefficient (0.5372) for saliency map generation on the MosMed dataset.

Limitations and Future Work

  • Dependency on a Baseline Model: The current ACAT framework requires a pre-trained baseline model to generate the initial saliency maps. While this is common in research, it adds a dependency. The authors suggest this approach can still fit within a normal research pipeline for comparison.
  • Generalizability to Other Tasks: The paper has been tested on brain and lung CT scans. The authors believe the approach can generalize to many other tasks and suggest further testing as future work.
  • Potential for Trade-off: While ACAT achieves the best accuracy and specificity, some other methods (like HSM) might offer better sensitivity. This suggests a potential trade-off, and the choice of method might depend on the specific application's priorities (e.g., minimizing false positives vs. false negatives).

Summary updated

BibTeX
@inproceedings{fontanella2023acat,
  author = {Alessandro Fontanella and Antreas Antoniou and Wenwen Li and Joanna M. Wardlaw and Grant Mair and Emanuele Trucco and Amos J. Storkey},
  title = {{ACAT: Adversarial Counterfactual Attention for Classification and Detection in Medical Imaging}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=yrVIUwRtzy},
}
Figure