Can Physician Judgment Enhance Model Trustworthiness? A Case Study on Predicting Pathological Lymph Nodes in Rectal Cancer

Kazuma Kobayashi, Yasuyuki Takamizawa, Mototaka Miyake, Sono Ito, Lin Gu, Tatsuya Nakatsuka, Yu Akagi, Tatsuya Harada, Yukihide Kanemitsu, Ryuji HamamotoArtif. Intell. Medicine 2023 · OpenReview

TL;DR

This paper investigates whether AI explanations, specifically attention maps from a multimodal transformer, can enhance physician trust and reduce uncertainty in predicting rectal cancer metastasis. Surprisingly, the study found that physician agreement with these explanations did not significantly reduce model uncertainty, suggesting that current attention mechanisms might offer superficial clarity without genuine trustworthiness enhancement.

What Problem Does This Paper Address?

The increasing integration of AI in healthcare hinges on physician trust, which in turn relies on explainability. However, the actual benefit of explainable AI (XAI) in clinical settings remains unclear. Specifically, there's a lack of robust evaluation frameworks to assess XAI's practical impact, limited physician-centered evaluations, and uncertainty about the utility of attention mechanisms in advanced models like transformers. This paper tackles these issues by proposing an evaluation framework that links explanation agreement with physician understanding to model uncertainty reduction, and then testing this framework on a real-world clinical task.

Methods

The core of this study involves a multimodal transformer model designed to predict pathological lymph node metastasis using both tabular clinical data and 3D MRI scans. For explainability, they employed "Transformer Explainability," a technique to visualize class-specific attention maps. To evaluate trustworthiness, they developed an "uncertainty estimator." This estimator quantifies model uncertainty in two ways: an "uninformed estimator" using only prediction probability variance, and an "informed estimator" that also incorporates an "agreement score." This agreement score is derived from how well the visualized attention maps align with expert physicians' understanding of the relevant clinical features. The key idea is to see if this physician-AI agreement, when fed into the informed estimator, can further reduce uncertainty compared to the uninformed estimator.

Datasets

The study utilized a dataset of 900 rectal cancer patients. This dataset was split into training, validation, and testing sets. A pre-training dataset was also created. The data included preoperative clinical information (demographics, tumor markers, etc.) and T2-weighted MRI scans. Postoperative pathological findings on lymph node metastasis served as the ground truth.

Key Insights and Ideas

  • Explainability's Double-Edged Sword: While explanations are crucial for trust, superficial clarity can be misleading. The study suggests that attention maps, while visually appealing, might not always translate to genuine understanding or reduced uncertainty.
  • Physician Agreement Isn't Enough: The core hypothesis was that if AI explanations align with physician judgment, it should lead to lower model uncertainty. This study found that this alignment, by itself, did not significantly improve uncertainty estimation.
  • Need for Deeper Evaluation: The findings highlight the limitations of current XAI evaluation methods and the necessity for more rigorous, human-centered assessments that go beyond mere visual agreement.
  • Uncertainty as a Trust Metric: The study reinforces the idea that quantifying model uncertainty is a vital component of building trustworthy AI, as it directly relates to the reliability of predictions.

Key Results

  • Model Performance: The multimodal transformer achieved moderate predictive performance, with an accuracy of 0.60 and an AUC-ROC of 0.68 on the testing dataset for predicting lymph node metastasis.
  • Agreement Scores: Expert physicians provided agreement scores for attention maps, indicating varying levels of alignment with their understanding. However, these scores showed no statistically significant correlation with the model's prediction correctness for either MRI or tabular data.
  • Uncertainty Estimation: Both the uninformed and informed uncertainty estimators showed a decreasing trend in cumulative accuracy and F1-score as uncertainty increased, indicating that lower uncertainty generally correlated with higher predictive correctness.
  • No Significant Benefit from Agreement: Crucially, the informed estimator, which included physician agreement scores, did not significantly outperform the uninformed estimator in reducing uncertainty or stratifying predictive performance (as shown by overlapping ROC curves in Fig. 8b). The correlation coefficient between the informed estimator's uncertainty and actual correctness (-0.28) was only slightly lower than that of the uninformed estimator (-0.30), failing to demonstrate a substantial benefit.

Limitations and Future Work

The authors acknowledge several limitations:

  • The study is based on data from a single facility.
  • The deep-learning model is limited to a transformer-based architecture.
  • The explainability technique is restricted to one specific post-hoc method (Transformer Explainability).
  • The predictive performance of the multimodal transformer is moderate, which might influence the observed effects of explainability.

Future work should focus on:

  • Developing more robust evaluation frameworks for XAI that go beyond superficial agreement.
  • Investigating ante hoc (explainability-by-design) approaches, which may offer more inherent trustworthiness.
  • Determining which specific attributes of explainability are truly beneficial for clinical decision-making.
  • Exploring how to design explanations that genuinely reduce uncertainty and prevent automation bias.

Summary updated

Abstract

Explainability is key to enhancing artificial intelligence's trustworthiness in medicine. However, several issues remain concerning the actual benefit of explainable models for clinical decision-making. Firstly, there is a lack of consensus on an evaluation framework for quantitatively assessing the practical benefits that effective explainability should provide to practitioners. Secondly, physician-centered evaluations of explainability are limited. Thirdly, the utility of built-in attention mechanisms in transformer-based models as an explainability technique is unclear. We hypothesize that superior attention maps should align with the information that physicians focus on, potentially reducing prediction uncertainty and increasing model reliability. We employed a multimodal transformer to predict lymph node metastasis in rectal cancer using clinical data and magnetic resonance imaging, exploring how well attention maps, visualized through a state-of-the-art technique, can achieve agreement with physician understanding. We estimated the model's uncertainty using meta-level information like prediction probability variance and quantified agreement. Our assessment of whether this agreement reduces uncertainty found no significant effect. In conclusion, this case study did not confirm the anticipated benefit of attention maps in enhancing model reliability. Superficial explanations could do more harm than good by misleading physicians into relying on uncertain predictions, suggesting that the current state of attention mechanisms in explainability should not be overestimated. Identifying explainability mechanisms truly beneficial for clinical decision-making remains essential.
BibTeX
@inproceedings{kobayashi2023can,
  author = {Kazuma Kobayashi and Yasuyuki Takamizawa and Mototaka Miyake and Sono Ito and Lin Gu and Tatsuya Nakatsuka and Yu Akagi and Tatsuya Harada and Yukihide Kanemitsu and Ryuji Hamamoto},
  title = {{Can Physician Judgment Enhance Model Trustworthiness? A Case Study on Predicting Pathological Lymph Nodes in Rectal Cancer}},
  booktitle = {Artif. Intell. Medicine},
  year = {2023},
  url = {https://openreview.net/forum?id=7uDqJTjkJgw},
}
Figure