IterMask3D: Unsupervised anomaly detection and segmentation with test-time iterative mask refinement in 3D brain MRI

Ziyun Liang, Xiaoqing Guo, Wentian Xu, Yasin Ibrahim, Natalie Voets, Pieter M. Pretorius, J. Alison Noble, Konstantinos KamnitsasMedical Image Analysis 2026 · DOI

Abstract

IterMask3D is a novel unsupervised framework designed for the detection and segmentation of anomalies in 3D brain MRI. By iteratively refining a spatial mask at test time, the method overcomes the fundamental sensitivity-precision trade-off inherent in standard "corrupt-and-reconstruct" models. The framework integrates high-frequency structural guidance to preserve normal anatomy and a subject-specific stopping threshold to adapt to individual patient data. Experimental results demonstrate significant improvements over state-of-the-art diffusion and autoencoder models in both synthetic artifact detection and real-world pathology segmentation.

What Problem Does This Paper Address?

Supervised models in medical imaging are highly effective for known pathologies but often fail to generalize to unseen diseases or unexpected imaging artifacts. Unsupervised Anomaly Detection (UAD) addresses this by learning the distribution of "normal" anatomy and flagging deviations.

The standard UAD approach follows a "corrupt-and-reconstruct" paradigm: an image is intentionally degraded (e.g., via noise or compression), and a model is trained to restore it. At test time, the model successfully restores normal tissue but fails to reconstruct anomalies, creating a high reconstruction error that serves as an anomaly mask. However, these methods suffer from a sensitivity-precision trade-off: excessive corruption leads to false positives in normal regions, while insufficient corruption allows the model to accidentally reconstruct the anomaly, leading to false negatives.

Comparison of noise-based and compression-based corruption in standard UAD.

Methodology

IterMask3D introduces three key innovations to break the sensitivity-precision deadlock:

Overview of the IterMask3D framework, including training, iterative refinement, and thresholding.
  1. Iterative Spatial Mask Refinement: Rather than applying global noise, the method begins by masking the entire brain. It reconstructs the masked area, identifies low-error regions (likely normal tissue), and progressively "unmasks" them. This iterative shrinking allows the model to leverage surrounding confirmed normal tissue to improve reconstruction accuracy for the remaining masked areas.
  2. High-Frequency Structural Guidance: To prevent the model from accidentally reconstructing anomalies, it is provided with a high-pass filtered version of the image. This "edge map" acts as a structural skeleton. Since the model is trained only on normal brains, it can use these edges to restore normal anatomy but lacks the generative knowledge to use anomalous edges (e.g., from a tumor) for reconstruction.
  3. Subject-Specific Thresholding: To handle varying intensity characteristics across patients, IterMask3D fits a tangent-based mathematical model ($\tau(t) = a \tan(bt)$) to the reconstruction error dynamics. The mask-shrinking process automatically stops when the error rate of change reaches a specific point, providing a truly automated, per-patient detection system.

Datasets and Evaluation

The model was trained on "normal" brain data from the OASIS3 (T2-weighted) and ADNI (FLAIR) datasets. Evaluation was conducted across a diverse range of anomalies:

  • Synthetic Artifacts: Gaussian noise, spike noise, bias fields, and ghosting.
  • Real-World Artifacts: Metallic susceptibility artifacts and failed skull-stripping cases.
  • Pathology: Gliomas from BraTS 2021 and stroke lesions from ISLES 2015.

Key Results and Insights

  • Superior Performance: IterMask3D achieved an AUROC of 100.0 on several synthetic artifacts, significantly outperforming standard Autoencoders (AE) and Denoising Diffusion Probabilistic Models (DDPM).
  • Pathology Segmentation: On the BraTS FLAIR dataset, the method achieved a Dice score of 69.7, a substantial improvement over standard DDPM (28.5) and the previous state-of-the-art DAE (60.0).
  • Robustness to "Black Holes": The method is particularly effective at identifying metallic artifacts that appear as signal voids. Unlike generative models that might try to "fill in" these holes with plausible-looking fake tissue, IterMask3D correctly identifies them as anomalies through the iterative refinement process.

Limitations and Future Work

While highly effective, IterMask3D still faces challenges with low-contrast lesions that have intensities very similar to normal gray matter. Additionally, significant domain shifts between training and testing scanners can still trigger false positives. Future research will explore bi-directional mask refinement (allowing expansion and shrinking) and the integration of this approach into automated clinical quality control pipelines.

Summary updated

BibTeX
@article{liang2026itermask,
  author = {Ziyun Liang and Xiaoqing Guo and Wentian Xu and Yasin Ibrahim and Natalie Voets and Pieter M. Pretorius and J. Alison Noble and Konstantinos Kamnitsas},
  title = {{IterMask3D: Unsupervised anomaly detection and segmentation with test-time iterative mask refinement in 3D brain MRI}},
  journal = {Medical Image Analysis},
  year = {2026},
  doi = {10.1016/j.media.2025.103763},
}
Figure