Beyond Lagrangian Methods for Robust Peak-Cost Constrained Reinforcement Learning
Its authors withdrew this paper, so there is no full text to read or summarize.
Lists
Only you see this
Abstract
We study robust peak-cost constrained reinforcement learning, where the goal is to maximize reward while bounding the largest cost incurred along a trajectory -- unlike the constrained MDP (CMDP), which only keeps the \emph{expected cumulative} cost below a threshold. We first show that the problem is exactly equivalent to a CMDP on a state augmented with the running maximum cost, which restores zero duality gap but requires history-dependent policies, yields a cost signal that is active only at a new peak, and makes worst-case evaluation conservative. We therefore keep the reachability-constrained (RCRL) formulation on the original state space, for which strong duality can fail, and optimize a surrogate rather than a Lagrangian, estimating worst-case values through an integral probability metric. Our main contribution is the first convergence analysis for this surrogate. We prove that every iterate stays $\epsilon$-safe from a safe start which the primal-dual-based approach can not guarantee, and that a stationary point of the exact surrogate is reached. Under a nondegeneracy condition, we further obtain a globally $\epsilon$-optimal and $\epsilon$-safe policy in $\mathcal O(\epsilon^{-3})$ iterations from a safe start, and $\mathcal O(\epsilon^{-4})$ otherwise. Experiments on continuous-control benchmarks, including RCRL, augmented CMDP and robust CMDP baselines, show that the method sustains reward while enforcing peak-cost safety under dynamics perturbations.
BibTeX
@misc{mukhopadhyay2027beyond,
author = {Shilpa Mukhopadhyay and Santosh Mohan Rajkumar and Sourav Ganguly and Om Dagadgunde and Honghao Wei and Debdipta Goswami and Arnob Ghosh},
title = {{Beyond Lagrangian Methods for Robust Peak-Cost Constrained Reinforcement Learning}},
year = {2027},
url = {https://openreview.net/forum?id=4pHTs5KPPL},
howpublished = {OpenReview},
note = {Submitted to ICLR 2027},
}Loading related content...