Learning Causal Alignment for Reliable Disease Diagnosis

Mingzhou Liu, Xinwei Sun, Ching-Wen Lee, Yu Qiao, Yizhou Wang

Abstract

Aligning the decision-making process of machine learning algorithms with that of experienced radiologists is crucial for reliable diagnosis. While existing methods have attempted to align their diagnosis behaviors to those of radiologists reflected in the training data, this alignment is primarily associational rather than causal, resulting in pseudo-correlations that may not transfer well. In this paper, we propose a causality-based alignment framework towards aligning the model's decision process with that of experts. Specifically, we first employ counterfactual generation to identify the causal chain of model decisions. To align this causal chain with that of experts, we propose a causal alignment loss that enforces the model to focus on causal factors underlying each decision step in the whole causal chain. To optimize this loss that involves the counterfactual generator as an implicit function of the model's parameters, we employ the implicit function theorem equipped with the conjugate gradient method for efficient estimation. We demonstrate the effectiveness of our method on two medical diagnosis applications, showcasing faithful alignment to radiologists.

Introduction: The Problem of Shortcut Learning

Deep learning models in medical imaging are notoriously prone to shortcut learning. A model might achieve high accuracy by identifying dataset-specific artifacts—such as hospital-specific watermarks, scanner types, or technician markers—rather than the actual pathology. This leads to high performance on training data but catastrophic failure in real-world clinical settings where these shortcuts are absent.

Existing "visual alignment" methods attempt to mitigate this by forcing models to attend to the same pixels as radiologists. However, these methods are primarily associational. A model might look at the correct region but for the wrong reasons. To ensure reliability, a model must be causally aligned: its predictions should change if and only if the underlying clinical features change.

Proposed Methodology: Causal Alignment

The authors introduce a hierarchical framework that mirrors the clinical diagnostic pipeline: Image → Attributes (e.g., margin, spiculation) → Diagnosis.

Overview of the causal alignment framework and hierarchical diagnostic pipeline.

1. Counterfactual (CF) Generation

To identify the features a model considers "causal," the framework generates a counterfactual image—the minimal modification to the original image required to flip the model's prediction. The difference between the original and CF image highlights the features the model is actually utilizing for its decision.

2. Causal Alignment Loss

The model is penalized if its identified causal factors fall outside the regions or attributes annotated by human experts. This ensures the model's internal logic matches expert reasoning.

3. Hierarchical Alignment

When attribute data is available (e.g., LIDC-IDRI), the model is trained to first predict clinical attributes and then the final diagnosis. Alignment loss is applied at both stages, ensuring the model uses correct imaging features for attributes and correct attributes for the final diagnosis.

4. Implicit Optimization

Training this framework is computationally challenging because the CF generator acts as an "inner" optimization loop. To calculate gradients through this loop, the authors utilize the Implicit Function Theorem (IFT) combined with the Conjugate Gradient method. This allows for efficient end-to-end training without the prohibitive memory requirements of storing large Hessian matrices.

Experimental Results

The method was evaluated on several benchmarks, including LIDC-IDRI (lung CT), CBIS-DDSM (mammography), and BraTS (brain MRI). To test robustness, the authors also introduced synthetic shortcuts (e.g., "+" or "−" symbols in image corners).

Qualitative and quantitative results showing superior alignment precision compared to baselines.

Key Findings:

  • Superior Alignment Precision: On the LIDC-IDRI dataset, the "Precision of CAM" (measuring alignment with expert masks) reached 0.751, a massive improvement over baselines like Ross et al. (0.034) and ICNN (0.068).
  • Robustness to Spurious Correlations: While baseline models were easily distracted by synthetic shortcuts, the proposed model maintained high diagnostic accuracy (72.2% on lung nodules) by focusing on clinical features like tumor margins and internal structures.
  • Interpretability: Qualitative visualizations show the model focusing tightly on relevant clinical features, providing "proof of work" that radiologists can verify.

Conclusion

By shifting from associational to causal alignment, this framework ensures that AI models learn the underlying mechanisms of disease rather than dataset-specific shortcuts. This approach significantly enhances the reliability, interpretability, and generalizability of AI-driven medical diagnosis, paving the way for safer integration into clinical workflows.

Summary updated

Abstract

Aligning the decision-making process of machine learning algorithms with that of experienced radiologists is crucial for reliable diagnosis. While existing methods have attempted to align their diagnosis behaviors to those of radiologists reflected in the training data, this alignment is primarily associational rather than causal, resulting in pseudo-correlations that may not transfer well. In this paper, we propose a causality-based alignment framework towards aligning the model's decision process with that of experts. Specifically, we first employ counterfactual generation to identify the causal chain of model decisions. To align this causal chain with that of experts, we propose a causal alignment loss that enforces the model to focus on causal factors underlying each decision step in the whole causal chain. To optimize this loss that involves the counterfactual generator as an implicit function of the model's parameters, we employ the implicit function theorem equipped with the conjugate gradient method for efficient estimation. We demonstrate the effectiveness of our method on two medical diagnosis applications, showcasing faithful alignment to radiologists.
BibTeX
@misc{liu2023learning,
  author = {Mingzhou Liu and Xinwei Sun and Ching-Wen Lee and Yu Qiao and Yizhou Wang},
  title = {{Learning Causal Alignment for Reliable Disease Diagnosis}},
  year = {2023},
  eprint = {2310.01766},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.01766},
}
Figure