Solving Sample-Level Out-of-Distribution Detection on 3D Medical Images

Daria Frolova, Anton Vasiliuk, Mikhail Belyaev, Boris ShirokikharXiv 2022 · DOI · arXiv

TL;DR

This paper introduces Image Histogram Features (IHF), a novel, DL-free method for detecting out-of-distribution (OOD) samples in 3D medical images. IHF achieves near-perfect performance by leveraging image intensity histograms as embeddings, significantly outperforming existing DL-based methods and even placing second in the MOOD Challenge 2022.

What Problem Does This Paper Address?

Deep learning models excel at tasks when the data they encounter during inference is similar to the data they were trained on. However, in critical applications like medical imaging, this assumption often breaks. Differences in patient demographics, scanner protocols, or even the presence of rare pathologies can lead to "out-of-distribution" (OOD) data. If a DL model encounters such data, its performance can degrade significantly, potentially leading to misdiagnoses. While OOD detection methods are well-established for 2D images, adapting them to 3D medical volumes is challenging due to computational complexity. Existing 3D OOD methods, often DL-based, struggle to achieve satisfactory results, even with synthetic OOD data, suggesting that current feature representations might not be sufficiently descriptive for large volumetric images.

Methods

The core of this paper is the Image Histogram Features (IHF) method. It's a two-step, DL-free approach:

  1. Histogram Embedding: For each 3D medical image (CT or MRI), a standard preprocessing pipeline is applied (interpolation, intensity clipping, min-max scaling). Then, an intensity histogram is computed from the preprocessed image. This histogram, represented as a vector of bin values, serves as the image's embedding. The number of bins ($m$) is the only hyperparameter.
  2. OOD Detection Algorithm: The histogram embeddings are then fed into a classical OOD detection algorithm. The authors primarily use Mahalanobis distance, which measures the distance of a test sample's embedding from the distribution of embeddings from the training (in-distribution) data. They also consider a "Min distance" approach as a fallback for smaller datasets.

This approach bypasses the need for complex DL models for feature extraction, relying instead on the inherent descriptive power of intensity distributions.

Datasets

The authors utilized a comprehensive set of publicly available 3D medical imaging datasets, creating a benchmark for OOD detection.

  • In-Distribution (ID) Datasets:
    • LIDC-IDRI (CT): Lung nodule segmentation task.
  • VS-Seg (MRI): Vestibular schwannoma segmentation task.
  • Out-of-Distribution (OOD) Datasets: These were carefully selected to represent various distribution shifts:
  • Covariate Shifts: Cancer500 (CT), CrossMODA ETZ (MRI) (different acquisition protocols/scanners).
  • Semantic Shifts: CT-ICH (CT), LiTS (CT), Medseg9 (CT), MIDRC (CT), NSCLC (CT), EGD (MRI) (different anatomies, pathologies, or populations).
  • Synthetic Anomalies: LIDC augm. (CT), VS-Seg augm. (MRI) (generated by applying transformations like contrast changes, noise, or blurring to ID data).

The paper also mentions their submission to the Medical Out-of-Distribution (MOOD) Challenge 2022, which uses a smaller, curated set of MR and CT images.

Key Insights and Ideas

  • Histograms are Powerful Embeddings for 3D Medical Images: The central idea is that simple intensity histograms capture sufficient information to distinguish between in-distribution and out-of-distribution 3D medical images, even for complex shifts. This challenges the notion that deep, learned features are always necessary.
  • DL-Based OOD Detection Struggles with 3D Medical Data: The paper highlights the significant underperformance of current DL-based OOD methods on 3D medical images, suggesting that their learned embeddings might not be robust to the diverse shifts encountered in this domain.
  • DL-Free Methods Can Be Superior: IHF's success demonstrates that simpler, model-free approaches can outperform complex DL models in specific OOD detection scenarios, especially when computational efficiency and interpretability are concerns.
  • MOOD Challenge Limitations: The authors argue that the MOOD challenge, by focusing heavily on synthetic anomalies, might inadvertently encourage methods that exploit specific anomaly generation techniques rather than developing generalizable OOD detection capabilities.

Key Results

  • Near-Perfect Performance: IHF achieved AUROC scores of 0.99 to 1.00 across most of the evaluated setups, indicating it misses less than 1% of OOD samples.
  • Outperforms State-of-the-Art: IHF consistently outperformed all other considered methods, including DL-based approaches like Ensemble, G-ODIN, and specialized 3D OOD methods like SVD and MOOD-top-1, by a significant margin.
  • MOOD Challenge Success: IHF placed second in the sample-level track of the MOOD Challenge 2022, demonstrating its effectiveness on a challenging, independently designed benchmark.
  • Scalability: The method showed strong performance across various OOD shifts (covariate, semantic, synthetic) and modalities (CT, MRI).
  • Other Applications: IHF embeddings also proved effective for domain shift detection (0.989 accuracy with an SVM classifier) and contrast detection in CT scans (0.87 accuracy).

Limitations and Future Work

  • Dataset Dependency: The authors acknowledge that OOD detection performance is heavily influenced by the specific datasets used. While their benchmark is extensive, other unconsidered image alternations exist.
  • Small Dataset Issue: For datasets with fewer samples than the number of histogram bins ($N_{tr} < m$), Mahalanobis distance becomes numerically unstable. In such cases, a "Min distance" approach is recommended, though it yields slightly lower performance.
  • Future Directions:
    • Regularization for Small Datasets: Developing techniques to regularize the sample covariance matrix or apply dimensionality reduction to embeddings to improve performance on small datasets.
  • Improving DL-Based Methods: Investigating how to develop DL models that better preserve domain-specific information and downstream task quality, addressing the observed mediocre performance of current DL methods.
  • Controlled Benchmarking: Creating benchmarks with disentangled and controlled sources of distribution shifts to better understand OOD detection method behavior.

Summary updated

Abstract

Deep Learning (DL) models tend to perform poorly when the data comes from a distribution different from the training one. In critical applications such as medical imaging, out-of-distribution (OOD) detection helps to identify such data samples, increasing the model's reliability. Recent works have developed DL-based OOD detection that achieves promising results on 2D medical images. However, scaling most of these approaches on 3D images is computationally intractable. Furthermore, the current 3D solutions struggle to achieve acceptable results in detecting even synthetic OOD samples. Such limited performance might indicate that DL often inefficiently embeds large volumetric images. We argue that using the intensity histogram of the original CT or MRI scan as embedding is descriptive enough to run OOD detection. Therefore, we propose a histogram-based method that requires no DL and achieves almost perfect results in this domain. Our proposal is supported two-fold. We evaluate the performance on the publicly available datasets, where our method scores 1.0 AUROC in most setups. And we score second in the Medical Out-of-Distribution challenge without fine-tuning and exploiting task-specific knowledge. Carefully discussing the limitations, we conclude that our method solves the sample-level OOD detection on 3D medical images in the current setting.
BibTeX
@misc{frolova2022solving,
  author = {Daria Frolova and Anton Vasiliuk and Mikhail Belyaev and Boris Shirokikh},
  title = {{Solving Sample-Level Out-of-Distribution Detection on 3D Medical Images}},
  year = {2022},
  eprint = {2212.06506},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.06506},
}
Figure