Beyond Ensemble Disagreement: Predictive STL Monitoring of RL Locomotion under Distribution Shift

Qurban Ali, Badaruddin Chachar, Maqsood Ali, Sajjad Ali RajperarXiv 2026 · arXiv

TL;DR

This paper investigates whether ensemble disagreement in world-model forecasts can serve as a reliable early warning for safety violations in RL locomotion under distribution shift. The authors find that while predicted robustness is a strong indicator of failure, ensemble disagreement is surprisingly uninformative—often performing no better than chance—and that simple reactive monitoring of the current state is nearly as effective as complex predictive rollouts.

What Problem Does This Paper Address?

Reinforcement Learning (RL) locomotion policies are typically trained in static simulations. When deployed in the real world, they encounter distribution shifts—such as sensor noise, varying terrain, or changes in physical parameters (like mass)—that can lead to catastrophic safety violations.

To prevent these, researchers use runtime monitors. A "reactive" monitor checks if the current state is safe, while a "predictive" monitor uses a learned world model to forecast the future and anticipate violations before they happen. However, under distribution shift, the world model itself becomes unreliable. A common heuristic is to use an ensemble of models and treat their disagreement as a proxy for uncertainty: if the models don't agree on the future, the situation is likely unsafe. This paper rigorously tests whether this "common wisdom" actually holds up for safety-critical locomotion tasks.

Methods

The authors developed a predictive monitoring pipeline based on Signal Temporal Logic (STL), which provides a mathematical framework to quantify how "safe" a trajectory is (its "robustness").

  1. Pipeline: At each timestep, the current observation is fed into a frozen PPO policy and an ensemble of three lightweight world models.
  2. Closed-Loop Rollouts: Each model generates a $K$-step future trajectory by repeatedly querying the policy on its own predicted observations.
  3. STL Evaluation: These predicted trajectories are evaluated against safety specifications (e.g., "don't fall," "maintain a certain height").
  4. Metrics: They compared Predicted Robustness ($P$) (the average safety score across the ensemble) against STL Disagreement ($U_\rho$) (the standard deviation of those scores). They also tested a Disagreement-Penalized Score ($S_{UP}$) that lowers the safety estimate when models disagree.
Figure 1: Predictive STL monitoring pipeline. The current observation ot feeds the frozen PPO policy and the world-model ensemble, whose members each generate a K-step closed-loop rollout by repeatedly querying the policy on predicted observations. Predicted trajectories are evaluated with STL and aggregated into predictive robustness P and STL disagreement Uρ; independently, O is evaluated on the observed trajectory τt:t+K, and SUP penalizes P by Uρ.

Datasets

The study utilized two standard MuJoCo locomotion environments:

  • Ant-v5: A 3D quadruped.
  • HalfCheetah-v5: A 2D planar locomotion system.

The models were trained on nominal data and then evaluated under several types of distribution shift:

  • External Forces: 5N and 15N pushes.
  • Observation Noise: Gaussian noise at varying intensities.
  • Physical Parameters: Scaling of mass and friction.
  • Compound Shifts: Combinations of the above.

Key Insights and Ideas

  • Disagreement is a Poor Proxy for Safety: The most striking finding is that ensemble disagreement ($U_\rho$) does not reliably track prediction error or future safety violations. In many cases, its ability to rank failures was no better than a coin flip (AUROC $\approx$ 0.5).
  • The Power of the Present: The authors found that simply monitoring the robustness of the current state (reactive monitoring) was nearly as effective as predicting the future. The predictive monitor only provided a marginal improvement (within 0.03 AUROC) over the reactive baseline.
  • Systematic Optimism: Under certain shifts, like severe observation noise, world models become systematically optimistic—they predict the robot will stay safe even when the actual trajectory is about to fail.
  • Penalization Fails: Adding a penalty for disagreement to the safety score did not consistently improve the ranking of failures, suggesting that disagreement doesn't provide "extra" information that isn't already captured by the predicted robustness itself.

Key Results

The quantitative evaluation focused on the "any-violation" target—whether any safety specification was violated within a $K$-step window.

  • Failure Ranking: Predicted robustness ($-P$) ranked future violations strongly, achieving AUROCs between 0.81 and 0.98 for HalfCheetah across various shifts.
  • Disagreement Performance: In contrast, STL disagreement ($U_\rho$) stayed near chance levels, with AUROCs between 0.47 and 0.56.
  • Horizon Sensitivity: As the prediction horizon ($K$) increased, violations became more frequent, but the disagreement signal did not become more informative.
Figure 2: HalfCheetah any-violation failure prediction (descriptive mean±std over 3 seeds). Left (K=10, Tab. 1): Uρ stays near chance (0.47–0.56) while −P ranks failures strongly (0.92–0.98); Uo/Uz are likewise near chance (0.44–0.54). Right (K=20, severe noise, 64.2±24.4% prevalence): Uρ/Uo/Uz give 0.65/0.47/0.46 versus 0.99 for −P. Dashed line marks chance.

Limitations and Future Work

  • Architectural Scope: The study used lightweight feedforward ensembles. The authors note that more complex recurrent or stochastic models (like the Dreamer architecture) might yield more informative disagreement, which remains an open question.
  • Environment Diversity: The evaluation was limited to two MuJoCo environments. The "Ant" environment proved difficult to benchmark because it was either almost always safe or almost always failing in the tested slices, leading to single-class data.
  • Sim-to-Real Gap: The study was entirely in simulation and did not account for real-world factors like latency, actuation noise, or complex contact dynamics.
  • Future Directions: The authors suggest exploring conformal calibration to provide formal safety guarantees and extending this predictive monitoring framework to real-world robotic hardware.

Summary updated

Abstract

RL locomotion policies trained under nominal simulation conditions may encounter distribution shifts that compromise safety. Predictive monitors could anticipate violations, but their world-model forecasts may become unreliable under shift, and ensemble disagreement is not calibrated. We evaluate predictive Signal Temporal Logic (STL) monitoring for frozen Ant-v5 and HalfCheetah-v5 policies using closed-loop world-model rollouts and a three-model ensemble. Across external-force, observation-noise, and mass shifts, predicted robustness ranks HalfCheetah's future STL violations strongly, whereas disagreement does not reliably track observation prediction error or violations. Under severe noise, violations remain common ($64.2\pm24.4\%$ at $K{=}20$ across three PPO seeds), yet current-state robustness without prediction comes within $0.03$ AUROC of predictive robustness. Penalizing predicted robustness by disagreement does not consistently improve ranking. Ant provides complementary per-specification evidence, but its composite target has no positive violations and its any-violation target is single-class in most evaluated seed-condition slices. Overall, predictive robustness provides useful STL-specific ranking but little demonstrated additional ranking value over the current state; these findings caution against treating ensemble disagreement as a reliable indicator of safety uncertainty.
BibTeX
@misc{ali2026beyond,
  author = {Qurban Ali and Badaruddin Chachar and Maqsood Ali and Sajjad Ali Rajper},
  title = {{Beyond Ensemble Disagreement: Predictive STL Monitoring of RL Locomotion under Distribution Shift}},
  year = {2026},
  eprint = {2611.02218},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02218},
}
Figure