Abstract
Off-policy Constrained Reinforcement Learning (CRL) offers high data efficiency but is often plagued by inaccurate cost estimation (C-function bias), leading to safety violations. We present Adaptive Ensemble C-learning (AEC), which utilizes a PID-controlled ensemble to dynamically correct estimation bias, and the First C Buffer (FCB) to align safety constraints with initial episode states. Our integrated algorithms, AECSAC and AECTQC, demonstrate superior performance and safety adherence across diverse robotics benchmarks compared to existing on-policy and off-policy methods.
Introduction
In Reinforcement Learning (RL), agents typically learn through a process of trial and error. However, in real-world robotics, "error" can result in physical damage or safety breaches. Constrained RL (CRL) addresses this by maximizing rewards while ensuring a predefined "cost" (e.g., collision impact or velocity) remains below a threshold.
The primary challenge in CRL is the trade-off between efficiency and accuracy. On-policy algorithms are generally safer but require vast amounts of data. Conversely, off-policy algorithms are data-efficient through experience reuse but struggle with accurate C-function estimation—the expected future cost. Fluctuations in the importance of safety (represented by the Lagrange multiplier $\beta$) during training lead to unpredictable over- or underestimation of costs, causing the agent to be either reckless or overly timid.
Methodology
To stabilize off-policy CRL, we introduce two key innovations: Adaptive Ensemble C-learning (AEC) and the First C Buffer (FCB).
Adaptive Ensemble C-learning (AEC)
AEC addresses cost estimation bias by employing an ensemble of $N$ C-functions. A PID controller dynamically adjusts a parameter $K$ to select ensemble statistics. When $K$ is positive, the algorithm averages the most pessimistic (highest) cost estimates; when negative, it averages the most optimistic (lowest) estimates. The paper mathematically proves (Theorem 1) that adjusting $K$ allows for a smooth transition from underestimation to overestimation bias. By comparing these estimates against periodic Monte Carlo "ground truth" simulations, the PID loop tunes $K$ in real-time to cancel out bias.
First C Buffer (FCB)
Standard CRL often estimates safety by averaging costs across an entire replay buffer. However, safety constraints are typically defined relative to the initial state of an episode. The FCB specifically stores and samples initial states, ensuring the agent's safety evaluations are mathematically congruent with the environment's defined thresholds.
Results and Discussion
We integrated these techniques into two frameworks: AECSAC (based on Soft Actor-Critic) and AECTQC (based on Truncated Quantile Critics, a distributional RL approach). Evaluation across nine robotics tasks—including MuJoCo speed limits, circle navigation, and Safety Gymnasium tasks—revealed several key insights:
- Superior Efficiency: AECSAC and AECTQC reached higher reward levels significantly faster than on-policy baselines like P3O or FOCOPS.
- Constraint Adherence: In complex environments like Humanoid-v3, our algorithms maintained cost budgets more effectively than other off-policy methods, which often failed to balance reward and safety.
- Stability: Ablation studies confirmed that the AEC component is critical; without it, learning curves exhibited massive fluctuations due to uncontrolled bias.
Conclusion
AEC and FCB provide a robust framework for off-policy CRL by directly addressing the instability of cost estimation. While the approach introduces some computational overhead due to ensemble maintenance and Monte Carlo simulations, the gains in data efficiency and safety adherence are substantial. Future work will focus on more efficient "true" C-value estimation to reduce the computational burden and further refine PID hyperparameter tuning.
Summary updated