Statistical testing on generative AI anomaly detection tools in Alzheimer's Disease diagnosis

Rosemary He, Ichiro TakeuchiarXiv 2024 · DOI · arXiv

Introduction

Generative AI (genAI) has demonstrated significant potential in medical imaging for detecting anomalies such as tumors or tissue atrophy. However, the clinical adoption of these "black box" models is hindered by the "double dipping" problem in hypothesis testing. In traditional statistics, hypotheses are defined a priori. In AI-based anomaly detection, the model identifies a suspicious region (the hypothesis) and tests its significance using the same data, leading to selection bias and high false positive rates (Type I errors). This research introduces a mathematically rigorous framework using Selective Inference (SI) to validate AI findings without requiring independent datasets.

Methodology

The proposed diagnostic pipeline integrates three key techniques:

1. Optical Flow Estimation

To capture the dynamic nature of neurodegeneration, the researchers utilize longitudinal MRI pairs taken at least one year apart. Optical flow algorithms track pixel movement over time to estimate the velocity of change. In these maps, dark pixels indicate atrophy (tissue loss), while light pixels represent growth, such as the expansion of brain ventricles.

Tracking brain changes over time using optical flow estimation.

2. Conditional Variational Autoencoder (CVAE)

A CVAE is trained exclusively on healthy subjects to learn a representation of normal brain morphology. When presented with a patient's image, the model reconstructs a "healthy" version. The discrepancy between the actual image and the reconstruction highlights potential anomalies.

3. Selective Inference (SI)

The core contribution is the application of Selective Inference. SI adjusts p-values by conditioning the statistical test on the selection event performed by the AI model. This approach accounts for selection bias, ensuring that the resulting significance scores are statistically valid.

Experimental Results

The study utilized data from the ADNI and OASIS databases, resulting in a cohort of 888 healthy and 110 diseased individuals.

Statistical Validity

A critical measure of a statistical test's validity is the distribution of p-values under the null hypothesis (no real effect), which should be uniform between 0 and 1. As shown in the comparison below, "naive" methods exhibit a distribution heavily skewed toward zero, leading to excessive false positives. In contrast, Selective Inference produces the expected uniform distribution.

Comparison of p-value distributions: Naive methods (left) show significant selection bias, while Selective Inference (right) maintains a uniform distribution under the null hypothesis.

Performance Metrics

The researchers compared SI against Naive and Bonferroni correction methods:

  • False Discovery Rate (FDR): In healthy samples, the Naive method had an FDR of 0.74. SI successfully controlled the FDR at 0.06 (target $\alpha = 0.05$).
  • Statistical Power: In the Alzheimer’s cohort, the conservative Bonferroni method detected anomalies in only 7% of cases. SI significantly improved detection power to 36% while remaining statistically valid.

Conclusion

This work demonstrates that Selective Inference provides a robust "middle ground" between the high false-positive rates of naive AI detection and the overly conservative nature of standard corrections like Bonferroni. By shifting the focus to dynamic biomarkers via optical flow, the method captures direct evidence of disease progression. Future work will focus on extending the framework to 3D volumes and optimizing anomaly thresholds to further enhance clinical utility.

Summary updated

BibTeX
@misc{he2024statistical,
  author = {Rosemary He and Ichiro Takeuchi},
  title = {{Statistical testing on generative AI anomaly detection tools in Alzheimer's Disease diagnosis}},
  year = {2024},
  eprint = {2410.13363},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.13363},
}
Figure