Adaptive Conformal Inference by Particle Filtering under Hidden Markov Models

Xiaoyi Su, Zhixin Zhou, Rui LuoarXiv 2024 · OpenReview

Abstract

Conformal inference is a statistical method used to construct prediction sets for point predictors, providing reliable uncertainty quantification with probability guarantees. This method utilizes historical labeled data to estimate the conformity or nonconformity between predictions and true labels. However, conducting conformal inference for hidden states under hidden Markov models (HMMs) presents a significant challenge, as the hidden state data is unavailable, resulting in the absence of a true label set to serve as a conformal calibration set. This paper proposes an adaptive conformal inference framework that leverages a particle filtering approach to address this issue. Rather than directly focusing on the unobservable hidden state, we innovatively use weighted particles as an approximation of the actual posterior distribution of the hidden state. Our goal is to produce prediction sets that encompass these particles to achieve a specific aggregate weight sum, referred to as the aggregated coverage level. The proposed framework can adapt online to the time-varying distribution of data and achieve the defined marginal aggregated coverage level in both one-step and multi-step inference over the long term. We verify the effectiveness of this approach through a real-time target localization simulation study.

Introduction

In safety-critical applications such as autonomous driving, robotics, and medical diagnostics, point estimates of a system's state are often insufficient. Reliable uncertainty quantification is required to understand the range of possible outcomes. Conformal Inference (CI) has emerged as a powerful tool for this purpose, offering distribution-free prediction sets with rigorous coverage guarantees.

However, a fundamental limitation of standard CI is its reliance on a "calibration set" of ground-truth labels. In systems modeled by Hidden Markov Models (HMMs), the true state is never directly observed; we only see noisy sensor data or "observations." This lack of ground truth makes it impossible to apply traditional CI techniques directly. This paper introduces a novel framework that bridges this gap by integrating Particle Filtering (PF) with Adaptive Conformal Inference (ACI).

Methodology

Particles as Proxies for Ground Truth

The core innovation of this research is the realization that while the true hidden state is unknown, the posterior distribution of that state can be approximated using a cloud of weighted particles. In a Particle Filter (PF) or Auxiliary Particle Filter (APF), these particles represent various hypotheses about the hidden state, each assigned a weight based on its likelihood given the observations.

Rather than attempting to calibrate against an unobservable point, the authors define aggregated coverage. The goal is to construct a prediction set such that the sum of the weights of all particles falling within that set meets a predefined confidence level (e.g., 90%). By treating the particle cloud as a surrogate for the true distribution, the framework provides a mathematically grounded way to quantify uncertainty in the absence of labels.

Adaptive Online Calibration

To handle non-stationary environments where data distributions may shift over time, the framework employs an adaptive update rule. The system calculates a conformity score based on the Euclidean distance between a point prediction and the particles. It then uses an online update strategy to adjust the quantile threshold ($\alpha_t$):

  1. Prediction: Generate a prediction set based on the current threshold.
  2. Evaluation: Measure the aggregated weight of particles within that set.
  3. Update: Adjust the threshold for the next time step based on the error (the difference between the achieved aggregated coverage and the target coverage).

This mechanism ensures that the prediction sets grow or shrink dynamically, maintaining the desired coverage level over the long term, even during multi-step-ahead forecasting.

Experimental Validation

The framework was evaluated using a real-time target localization simulation. The scenario involved a 2D target moving with constant velocity but subject to random Gaussian acceleration. A network of binary sensors provided "pings" based on the target's proximity. The simulation tracked trajectories over 1,000 time steps, utilizing 1,000 particles for the filtering process.

Key Results

  • Reliable Coverage: When the target coverage was set to 90%, the adaptive PF and APF methods achieved empirical coverage levels of 0.8966 and 0.8971, respectively. This demonstrates near-perfect alignment with the theoretical goals.
  • Conservative Safety Margin: Interestingly, the "Actual Coverage"—the frequency with which the true (simulated) hidden position was contained within the prediction set—was approximately 93-94%. This suggests the method is slightly conservative, providing an inherent safety margin.
  • Robustness to Drift: Unlike non-adaptive models, which fail to maintain coverage as conditions change, the adaptive mechanism effectively prevented confidence drift, even as the time horizon for predictions increased.

Conclusion and Future Work

This paper provides a robust solution for uncertainty quantification in HMMs by treating particle clouds as a valid proxy for unobservable states. The integration of ACI allows the system to remain accurate in dynamic environments.

Future research will focus on optimizing the adaptation parameter ($\gamma$), which currently requires manual tuning. The authors aim to develop a "dynamically-tuned" ACI that can automatically determine the optimal rate of adaptation. Additionally, efforts will be made to reduce the computational complexity of the framework to make it more suitable for high-dimensional state spaces in real-time applications.

Summary updated

Abstract

Conformal inference is a statistical method used to construct prediction sets for point predictors, providing reliable uncertainty quantification with probability guarantees. This method utilizes historical labeled data to estimate the conformity or nonconformity between predictions and true labels. However, conducting conformal inference for hidden states under hidden Markov models (HMMs) presents a significant challenge, as the hidden state data is unavailable, resulting in the absence of a true label set to serve as a conformal calibration set. This paper proposes an adaptive conformal inference framework that leverages a particle filtering approach to address this issue. Rather than directly focusing on the unobservable hidden state, we innovatively use weighted particles as an approximation of the actual posterior distribution of the hidden state. Our goal is to produce prediction sets that encompass these particles to achieve a specific aggregate weight sum, referred to as the aggregated coverage level. The proposed framework can adapt online to the time-varying distribution of data and achieve the defined marginal aggregated coverage level in both one-step and multi-step inference over the long term. We verify the effectiveness of this approach through a real-time target localization simulation study.
BibTeX
@misc{su2024adaptive,
  author = {Xiaoyi Su and Zhixin Zhou and Rui Luo},
  title = {{Adaptive Conformal Inference by Particle Filtering under Hidden Markov Models}},
  year = {2024},
  eprint = {2411.01558},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.01558},
}
Figure