Abstract
Reinforcement Learning (RL) agents often fail when deployed in environments that differ slightly from their training simulations. This paper introduces two frameworks for "Action Robust" RL—Probabilistic Action Robust MDP (PR-MDP) and Noisy Action Robust MDP (NR-MDP)—which model adversarial interference at the action level. By training agents against an adversarial "disturber," the authors develop policies that are significantly more resilient to model mismatches, such as changes in mass or friction. Experimental results on MuJoCo benchmarks demonstrate that these action-robust agents not only generalize better but often achieve higher peak performance than standard agents in noise-free settings.
Introduction
Standard Reinforcement Learning (RL) algorithms typically operate under the assumption that the training environment perfectly matches the deployment environment. In practice, this "perfect world" assumption is frequently violated due to factors like sensor noise, mechanical wear, or modeling inaccuracies in simulation. For instance, a robot trained to walk in a simulator may encounter different friction or mass distributions in the real world.
While traditional Robust MDP frameworks address uncertainty in transition dynamics, they are often mathematically complex and computationally expensive for high-dimensional continuous control tasks. This paper proposes a shift in focus from environment dynamics to action execution, asking: How can we train a policy that remains effective even when the executed action isn't exactly what the agent intended?
Methodology
The authors formalize two distinct criteria for action robustness:
- Probabilistic Action Robust MDP (PR-MDP): With a probability alpha, an adversary takes full control and executes the worst possible action. This models sudden, discrete interruptions.
- Noisy Action Robust MDP (NR-MDP): The final action is a weighted average of the agent’s choice and an adversary’s choice: $a_{final} = (1-alpha)a_{agent} + alpha a_{adversary}$. This models continuous, interrupting forces like wind or miscalibration.
To solve these, the authors introduce AR-DDPG (Action-Robust Deep Deterministic Policy Gradient). This approach frames RL as a zero-sum game where an "Actor" network maximizes rewards while a simultaneous "Adversary" network minimizes them. A "Critic" network estimates the value of states under this joint adversarial policy.
Experimental Evaluation
The proposed methods were evaluated using the MuJoCo physics engine across various locomotion (Hopper, Walker2d, Humanoid) and classic control (InvertedPendulum) tasks. Robustness was tested by evaluating trained policies in environments where physical parameters, specifically the mass of the robot, were varied significantly from the training setup.
Key Insights
- Implicit Regularization: Training with an adversary acts as a form of regularization, preventing the agent from overfitting to specific simulator trajectories.
- Enhanced Exploration: In PR-MDP, the adversary forces the agent to explore states it might otherwise avoid, leading to a more robust understanding of the environment.
- Simplicity: The approach requires only a single scalar parameter (alpha) to define robustness, avoiding the need for complex uncertainty sets.
Results
The Action-Robust agents demonstrated superior generalization compared to standard DDPG baselines. For example, in the Hopper-v2 task, robust agents maintained high performance even when the robot's mass was doubled or halved. Furthermore, in several domains like Walker2d and Humanoid, robust agents achieved higher peak rewards than the baseline even in the absence of noise.
One notable failure case was InvertedPendulum, where a sufficiently strong adversary could prevent the agent from ever learning to balance, making the task impossible.
Conclusion and Future Work
Action-robust frameworks provide a simple yet effective means of training RL agents for real-world deployment. While the NR-MDP implementation is technically a heuristic (as it may require stochastic policies for optimality), it shows strong empirical performance. Future work could explore state-dependent uncertainty (alpha(s)) to allow for more nuanced robustness in critical regions of the state space.
Summary updated