Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimation

Mohit Prashant, Arvind Easwaran, Suman Das, Michael YuhasAAAI 2025 · DOI

Abstract

Deep Reinforcement Learning (RL) agents often fail when encountering environments that differ from their training data. This paper introduces COTD (CVAE-based OOD Transition Detector), a framework that provides statistical guarantees for Out-Of-Distribution (OOD) detection. By leveraging ensembles of Conditional Variational Autoencoders (CVAEs) to model transition dynamics and applying Inductive Conformal Prediction (ICP), COTD identifies unfamiliar states and environment shifts with a user-defined confidence level. Experimental results across standard benchmarks show that COTD outperforms existing state-of-the-art methods by up to 17% in detection accuracy.

Introduction

Deep RL has shown remarkable success in simulated environments, yet its real-world deployment remains hindered by brittleness. An agent trained under specific conditions—such as a specific gravity or friction level—may fail catastrophically when these parameters shift slightly. The fundamental challenge lies in distinguishing between epistemic uncertainty (lack of exploration) and domain shift (changes in environment rules). Current OOD detection methods often lack formal guarantees, making them unreliable for safety-critical applications. This work addresses this gap by formalizing OOD execution and proposing a detection method with provable reliability.

The COTD Framework

The core innovation of COTD is its focus on transition estimation rather than value-based uncertainty. By monitoring the "physics" of the environment, the detector remains independent of the agent's specific policy.

Overview of the COTD architecture, featuring CVAE ensembles for state reconstruction and OOD determination.

The framework consists of three primary components:

  1. CVAE Ensembles: An ensemble of $N$ CVAEs is trained to reconstruct the next state $s_2$ given the current state $s_1$ and action $a_1$. High reconstruction error across the ensemble indicates an unfamiliar transition.
  2. Inductive Conformal Prediction (ICP): To provide statistical guarantees, COTD employs ICP. It uses a calibration dataset from the training environment to establish a dynamic threshold for reconstruction errors. This allows for a guaranteed true positive rate for OOD detection based on a confidence parameter $\delta$.
  3. Dual OOD Detection: The system distinguishes between State-Based OOD (reaching rare states) and Environment-Based OOD (changes in the underlying transition function, such as altered gravity).

Experimental Evaluation

The researchers evaluated COTD on four OpenAI Gym benchmarks: LunarLander, Ant, CartPole, and Pendulum. OOD scenarios were generated by modifying physical parameters (e.g., mass, wind power, turbulence) or introducing action perturbations.

Comparison of COTD against baseline models across multiple environments, showing superior AUC performance.

As shown in the results table, COTD consistently achieves the highest Overall AUC across all benchmarks. Notably, in the LunarLander environment, COTD improved detection performance by at least 17% over baselines like PEDM and DEXTER when environment parameters were altered. In the Ant environment, it maintained a high State AUC of 0.936, demonstrating its robustness in identifying rare states even when the environment rules remained constant.

Discussion and Limitations

By prioritizing transition dynamics over reward-based signals, COTD offers a more fundamental approach to OOD detection. The integration of Conformal Prediction transforms OOD detection from a heuristic-based task into a statistically rigorous process suitable for safety engineering.

However, several limitations remain. The current implementation assumes Gaussian transition distributions, which may not hold in highly complex or multi-modal environments. Additionally, the computational cost of running an ensemble of CVAEs in real-time can be significant. Future research will explore optimizing the trade-off between detection speed and accuracy, particularly in continuous-time Markov Decision Processes (MDPs) where state changes may be subtle over short intervals.

Summary updated

BibTeX
@inproceedings{prashant2025guaranteeing,
  author = {Mohit Prashant and Arvind Easwaran and Suman Das and Michael Yuhas},
  title = {{Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i12.33357},
}
Figure