TL;DR
This paper investigates how to bring reliable uncertainty estimation and out-of-distribution (OOD) state detection to on-policy actor-critic reinforcement learning (specifically PPO). The authors formalize value, policy, and multiplicative uncertainty measures, show that standard deep ensembles struggle due to the on-policy nature of PPO, and demonstrate that Masksembles provide an effective balance between high task reward and robust OOD detection.
What Problem Does This Paper Address?
Deep RL policies are notoriously brittle: when deployed in conditions that differ even slightly from training, they often fail silently without signaling distress. While uncertainty estimation and OOD detection are widely studied in supervised learning and off-policy Q-learning, they remain largely unexplored in on-policy actor-critic algorithms like Proximal Policy Optimization (PPO).
The authors identify two core reasons for this gap:
- Ambiguous definitions: In on-policy RL, data distributions continuously shift during training, and catastrophic forgetting means states seen early on can become OOD for the converged policy. Furthermore, standard supervised uncertainty metrics (like prediction entropy or maximum softmax probability) capture environment stochasticity (aleatoric uncertainty) or policy entropy bonuses rather than true epistemic uncertainty.
- The "On-Policy Dilemma": Multi-sample uncertainty estimators (such as ensembles or dropout) require diverse sub-models. In on-policy RL, however, diverse sub-models generate diverse trajectories, pushing training data off-policy and degrading learning stability.
Methods
- Formulating Uncertainty Metrics: The authors separate epistemic uncertainty into:
- Policy Uncertainty ($U^\Pi_{std}$): Measured as the disagreement (standard deviation) between sub-model policy outputs (action logits/probabilities for discrete actions, or predicted Gaussian means for continuous actions).
- Value Uncertainty ($U^V_{std}$): Measured as the standard deviation of value function predictions $V(s)$.
- Multiplicative Uncertainty ($U^{Mult}_{std}$): Defined as $U^\Pi_{std}(s) \times U^V_{std}(s)$, serving as an integrated state-action uncertainty measure.
- Architectural Implementations in PPO: They implement drop-in uncertainty estimation approaches using 4 sub-models/samples: Monte-Carlo Dropout (MCDO), Monte-Carlo Dropconnect (MCDC), Deep Ensembles, and Masksembles. Masksembles use a fixed set of overlapping binary masks, allowing explicit hyperparameter tuning of sub-model correlation while executing in a single, fast forward pass.
- Experience Synchronization: Sub-models are assigned to disjoint subsets of parallel vectorized environments, but their experiences are aggregated into a shared rollout buffer to compute synchronized updates.
- Bi-Objective Optimization: The authors run Pareto meta-optimization (via Optuna) to balance cumulative episode reward against OOD classification performance (ROC-AUC).
Datasets
- MuJoCo Continuous Control: Ant-v3, HalfCheetah-v3, Walker2d-v3, and Swimmer-v3. In-distribution (ID) data comes from nominal rollouts; OOD data is generated via randomized physical perturbations (varying gravity, wind, surface friction, and limb scaling factors).
- Atari Discrete Control: Seaquest-v4, MsPacman-v4, Berzerk-v4, and SpaceInvaders-v4. ID states are sampled from trained agent runs; OOD states are drawn from expert human demonstrations in advanced, unseen levels from the Atari-HEAD dataset.
- Synthetic Observation Noise: Uniform noise added to inputs during meta-hyperparameter tuning as a fast OOD proxy.
Key Insights and Ideas
- Ensembles fail without synchronization: Training independent ensemble members causes each policy to visit completely different trajectories. The models diverge, rendering ensemble averaging at inference incoherent. Even with a shared buffer, ensembles struggle to achieve competitive returns in PPO.
- Multiplicative uncertainty wins: Value uncertainty effectively flags unfamiliar states or physical instability (such as the robot levitating at spawn). Combining value and policy uncertainty multiplicatively consistently yields the highest OOD detection AUC.
- Masksembles resolve the diversity trade-off: By controlling mask overlap, Masksembles let practitioners tune the correlation between sub-models. This preserves the on-policy stability needed for high reward while retaining enough sub-model disagreement to identify OOD states.
Key Results
- Reward Preservation: Masksembles come closest to baseline PPO rewards (e.g., Ant-v3: 4884.8 vs. baseline 5189.7; Seaquest-v4: 3524.0 vs. baseline 3898.6), vastly outperforming Deep Ensembles (Ant-v3: 989.2; Seaquest-v4: 122.6).
- OOD Detection Performance: Using multiplicative uncertainty, Masksembles consistently lead OOD detection across tasks (ROC-AUC of 0.87 on HalfCheetah, 0.77 on Ant, 0.90 on Swimmer, 0.93 on MsPacman, and 0.85 on Seaquest).
- Model Scaling: Unlike Dropout or Ensembles, increasing the MLP hidden layer size specifically boosts the task reward of Masksembles, offsetting capacity lost to binary masking.
Limitations and Future Work
- Performance Gap: While Masksembles outperform competing uncertainty estimators, they still exhibit slight reward drop-offs compared to unmasked baseline PPO networks on certain tasks.
- Algorithm Specificity: Findings are centered primarily on PPO. Preliminary experiments on TRPO and A2C show that algorithms lacking PPO's clipped objective suffer more severely from sub-model policy divergence.
- Downstream Deployment: The authors note that leveraging these uncertainty metrics during training (for exploration) or during deployment (to trigger fallback controllers or query human assistance) remains an open avenue.
Summary updated