Abstract
Deep Reinforcement Learning (RL) agents often fail when encountering environments that differ from their training data. This paper introduces COTD (CVAE-based OOD Transition Detector), a framework that provides statistical guarantees for Out-Of-Distribution (OOD) detection. By leveraging ensembles of Conditional Variational Autoencoders (CVAEs) to model transition dynamics and applying Inductive Conformal Prediction (ICP), COTD identifies unfamiliar states and environment shifts with a user-defined confidence level. Experimental results across standard benchmarks show that COTD outperforms existing state-of-the-art methods by up to 17% in detection accuracy.
Introduction
Deep RL has shown remarkable success in simulated environments, yet its real-world deployment remains hindered by brittleness. An agent trained under specific conditions—such as a specific gravity or friction level—may fail catastrophically when these parameters shift slightly. The fundamental challenge lies in distinguishing between epistemic uncertainty (lack of exploration) and domain shift (changes in environment rules). Current OOD detection methods often lack formal guarantees, making them unreliable for safety-critical applications. This work addresses this gap by formalizing OOD execution and proposing a detection method with provable reliability.
The COTD Framework
The core innovation of COTD is its focus on transition estimation rather than value-based uncertainty. By monitoring the "physics" of the environment, the detector remains independent of the agent's specific policy.
The framework consists of three primary components:
- CVAE Ensembles: An ensemble of $N$ CVAEs is trained to reconstruct the next state $s_2$ given the current state $s_1$ and action $a_1$. High reconstruction error across the ensemble indicates an unfamiliar transition.
- Inductive Conformal Prediction (ICP): To provide statistical guarantees, COTD employs ICP. It uses a calibration dataset from the training environment to establish a dynamic threshold for reconstruction errors. This allows for a guaranteed true positive rate for OOD detection based on a confidence parameter $\delta$.
- Dual OOD Detection: The system distinguishes between State-Based OOD (reaching rare states) and Environment-Based OOD (changes in the underlying transition function, such as altered gravity).
Experimental Evaluation
The researchers evaluated COTD on four OpenAI Gym benchmarks: LunarLander, Ant, CartPole, and Pendulum. OOD scenarios were generated by modifying physical parameters (e.g., mass, wind power, turbulence) or introducing action perturbations.
As shown in the results table, COTD consistently achieves the highest Overall AUC across all benchmarks. Notably, in the LunarLander environment, COTD improved detection performance by at least 17% over baselines like PEDM and DEXTER when environment parameters were altered. In the Ant environment, it maintained a high State AUC of 0.936, demonstrating its robustness in identifying rare states even when the environment rules remained constant.
Discussion and Limitations
By prioritizing transition dynamics over reward-based signals, COTD offers a more fundamental approach to OOD detection. The integration of Conformal Prediction transforms OOD detection from a heuristic-based task into a statistically rigorous process suitable for safety engineering.
However, several limitations remain. The current implementation assumes Gaussian transition distributions, which may not hold in highly complex or multi-modal environments. Additionally, the computational cost of running an ensemble of CVAEs in real-time can be significant. Future research will explore optimizing the trade-off between detection speed and accuracy, particularly in continuous-time Markov Decision Processes (MDPs) where state changes may be subtle over short intervals.
Summary updated