Conformal Prediction with Cellwise Outliers: A Detect-then-Impute Approach

Qian Peng, Yajie Bao, Haojie Ren, Zhaojun Wang, Changliang Zou

TL;DR

This paper introduces a novel "detect-then-impute" framework for conformal prediction that robustly handles cellwise outliers in test data. By first identifying and imputing outlier entries in the test features, the method ensures valid prediction intervals even when data distributions shift due to contamination, a common issue in real-world applications.

What Problem Does This Paper Address?

Machine learning models are increasingly used in critical decision-making, necessitating reliable uncertainty quantification. Conformal prediction (CP) offers a powerful way to provide finite-sample coverage guarantees for prediction intervals (PIs). However, a key assumption of CP is data exchangeability, which is violated when test data features contain "cellwise outliers" – individual entries that deviate significantly from the norm. These outliers can arise from measurement errors or natural variations and compromise the reliability of PIs. Existing CP methods struggle with this, as the distribution shift caused by outliers is often unknown and difficult to estimate. This paper tackles the challenge of providing valid PIs in the presence of such cellwise outliers in the test set.

Methods

The core of the proposed approach is a "detect-then-impute" strategy.

  1. Detection (D): An outlier detection procedure is applied to the test feature to identify potentially contaminated entries.
  2. Imputation (I): The identified outlier entries are then imputed using a chosen imputation method.

To maintain the exchangeability required for CP, the authors adapt this detect-then-impute procedure to the calibration set as well. This results in processed features that are exchangeable with the processed test feature.

Two specific algorithms are developed:

  • PDI-CP (Proxy Detection-Imputation CP): This method uses the detected outliers in the test set as a proxy for the true outlier locations and applies imputation. It provides approximate marginal coverage.
  • JDI-CP (Joint Detection-Imputation CP): This algorithm uses a "joint mask" that considers both detected outliers in the test set and detected outliers in the calibration set. It leverages the Jackknife+ technique to achieve a finite-sample $1-2\alpha$ coverage guarantee.

The framework is designed to be flexible, allowing integration with various existing detection and imputation methods.

Datasets

The paper evaluates its methods on:

  • Synthetic datasets: Generated under different noise distributions (light-tailed, heavy-tailed, skewed) and contamination levels.
  • Real-world datasets:
    • Airfoil dataset: From the UCI Machine Learning Repository.
  • Wind direction dataset: From a meteorological station in Brazil.
  • Riboflavin dataset: A gene expression dataset.

For the real-world datasets, artificial cellwise outliers were introduced to simulate contamination.

Key Insights and Ideas

  • Cellwise outliers break standard CP: The paper clearly demonstrates that standard CP methods fail to provide valid coverage when test data contains cellwise outliers, highlighting the need for specialized approaches.
  • "Detect-then-impute" is a viable strategy: The core idea of identifying and correcting outliers before applying CP is shown to be effective.
  • Adaptation to calibration set is crucial: Simply applying detection and imputation to the test set is insufficient. Adapting these procedures to the calibration set is key to restoring exchangeability and ensuring valid coverage.
  • JDI-CP offers strong theoretical guarantees: The JDI-CP algorithm, by using a joint masking strategy and Jackknife+, achieves a finite-sample $1-2\alpha$ coverage guarantee, which is a significant theoretical contribution.
  • Robustness across methods: The proposed framework shows robust performance when combined with different detection (DDC, SVM, cellMCD) and imputation (Mean Imputation, kNN, MICE) methods, demonstrating its general applicability.

Key Results

  • Robust Coverage: The proposed PDI-CP and JDI-CP methods consistently achieve coverage close to the target level ($1-\alpha$ or $1-2\alpha$) across various synthetic and real-world scenarios, outperforming standard SCP and WCP methods, which often fail to reach the target coverage.
  • Comparable PI Lengths: The PIs produced by the proposed methods are comparable in length to an "oracle" baseline (which assumes knowledge of true outliers), indicating efficiency.
  • Theoretical Guarantees: JDI-CP is proven to provide a finite-sample $1-2\alpha$ coverage guarantee under the "sure detection" assumption for the outlier detector. PDI-CP provides approximate marginal coverage.
  • Performance under varying contamination: The methods maintain good coverage control even with increasing contamination levels and different detection thresholds.

Limitations and Future Work

  • Sure Detection Assumption: The theoretical guarantee for JDI-CP relies on the "sure detection" property of the outlier detector (Assumption 3.1), meaning all true outliers must be detected. While the paper discusses how to satisfy this in practice (e.g., by choosing smaller detection thresholds), it's a practical consideration.
  • Coverage Gap: PDI-CP's coverage gap depends on the imputation error and the number of false positives from the detector.
  • Future Work: The paper suggests exploring the combination of their framework with other non-conformity scores (like CQR) and investigating its performance under different types of data shifts beyond cellwise outliers. The paper also mentions an alternative, more conservative version of JDI-CP (C-JDI) that achieves $1-\alpha$ coverage but results in wider PIs.

Summary updated

BibTeX
@inproceedings{peng2025conformal,
  author = {Qian Peng and Yajie Bao and Haojie Ren and Zhaojun Wang and Changliang Zou},
  title = {{Conformal Prediction with Cellwise Outliers: A Detect-then-Impute Approach}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=saqVVehEOM},
}
Figure