Abstract
Deep Reinforcement Learning (RL) faces significant challenges in real-world deployment due to its sensitivity to environmental changes and safety constraints. This paper presents Optimal Transport Perturbations (OTP), a framework for Robust Constrained Markov Decision Processes (RC-MDPs). By utilizing Optimal Transport to define uncertainty sets, OTP allows agents to learn robust policies in a safe, nominal environment by simulating worst-case transitions. Experimental results on the Real-World RL Suite demonstrate that OTP significantly improves safety and performance in perturbed environments compared to existing baselines.
Introduction
The deployment of Deep RL in physical systems is often hindered by the "sim-to-real" gap and the risk of safety violations. Traditional robust RL methods frequently rely on Domain Randomization, which requires precise knowledge of environmental parameters, or adversarial training, which can be physically hazardous. This work introduces a novel approach that decouples training from physical disturbances by applying "virtual" perturbations to collected data, providing formal robustness guarantees without the need for complex simulators or risky interactions.
Methodology
The OTP framework is built upon the concept of Robust Constrained MDPs, where the goal is to find a policy that maximizes reward while satisfying safety constraints under the worst-case transition model within an uncertainty set.
Optimal Transport Uncertainty Sets
The authors propose using the Optimal Transport (OT) cost, or Wasserstein distance, to define the uncertainty set around a nominal transition model $\hat{p}_{s,a}$. Unlike the Kullback-Leibler (KL) divergence, the OT distance considers the underlying metric of the state space, making it more suitable for modeling physical perturbations where states may shift continuously.
As illustrated in the figure above, the OT framework quantifies the distance between distributions by the minimum cost required to transport probability mass. This allows the uncertainty set to encompass transition models that are "close" in a physical sense, even if their mathematical supports do not perfectly overlap.
Virtual Worst-Case Transitions and Perturbation Networks
A key theoretical result of this work is the equivalence between finding the worst-case transition model in the OT uncertainty set and applying a deterministic perturbation function to the state space.
This equivalence enables the implementation of a Perturbation Network. During the learning phase, this network takes a transition $(s, a, \hat{s}')$ from the nominal environment and generates a "virtual" next-state $s'$ that represents a worst-case scenario for either reward ($g_{s,a}^r$) or safety ($g_{s,a}^c$). The agent then updates its value functions based on these hallucinated worst-case transitions, effectively training for robustness in a purely "virtual" adversarial setting.
Experimental Results
The OTP framework was evaluated using the Real-World RL Suite (RWRL) on tasks including Cartpole, Walker, and Quadruped. These tasks were subjected to various perturbations such as changes in mass, length, and density.
The results demonstrate the superior robustness of OTP:
- Safety Compliance: OTP-trained policies maintained safety in 87% of perturbed test cases, a significant improvement over the 51% achieved by standard Safe RL.
- Risk Mitigation: The safety costs incurred by OTP were only 0.34x those of the baseline.
- Performance Gains: Despite its conservative nature, OTP achieved 1.06x higher rewards on average in perturbed environments, proving that robustness does not necessarily come at the cost of performance.
Conclusion
Optimal Transport Perturbations offer a robust and safe alternative to traditional RL training methods. By leveraging the geometric properties of Optimal Transport, the OTP framework provides a principled way to prepare agents for unknown environmental disturbances while maintaining safety in the training environment. Future research will focus on optimizing the uncertainty radius and extending the framework to more complex, stochastic real-world scenarios.
Summary updated