Abstract
Learning reward functions from data is a promising path towards achieving scalable Reinforcement Learning (RL) for robotics. However, a major challenge in training agents from learned reward models is that the agent can learn to exploit errors in the reward model to achieve high reward behaviors that do not correspond to the intended task. These reward delusions can lead to unintended and even dangerous behaviors. On the other hand, adversarial imitation learning frameworks tend to suffer the opposite problem, where the discriminator learns to trivially distinguish agent and expert behavior, resulting in reward models that produce low reward signal regardless of the input state. In this paper, we connect these two classes of reward learning methods to positive-unlabeled (PU) learning, and we show that by applying a large-scale PU learning algorithm to the reward learning problem, we can address both the reward under- and over-estimation problems simultaneously. Our approach drastically improves both GAIL and supervised reward learning, without any additional assumptions.
Introduction
Training robots via Reinforcement Learning (RL) typically requires a well-defined reward function. However, hand-coding these functions for complex tasks is notoriously difficult. Researchers often rely on two alternatives: supervised reward learning from human ratings and Adversarial Imitation Learning (GAIL).
Both approaches face significant hurdles. Supervised models often suffer from "reward delusions," where an agent discovers a state that the model incorrectly assigns a high reward, leading to "reward hacking." Conversely, GAIL often fails due to discriminator overfitting; the model learns to distinguish expert and agent behavior based on irrelevant environmental artifacts (like lighting or background objects) rather than task-relevant features, resulting in a vanishing reward signal.
Reframing Reward Learning as PU Learning
The authors argue that these failures stem from a common flawed assumption: treating all agent experience as "negative" data. In reality, as an agent improves, its experience becomes a mixture of successes and failures. Forcing a model to treat these emerging successes as negative data causes the discriminator to overfit to irrelevant differences.
Positive-Unlabeled Reward Learning (PURL) reframes the problem. It treats expert data as Positive and the agent's replay buffer as Unlabeled (a mixture of unknown proportions of success and failure). By using a non-negative PU risk estimator, the model is forced to identify features that truly define success rather than just finding any arbitrary difference between two datasets.
Methods
The researchers introduced two primary implementations:
- nn-PUGAIL: An adaptation of GAIL that uses a PU objective for the discriminator, preventing the risk estimate from becoming negative—a common failure mode in deep PU learning.
- nn-PURL: A semi-supervised approach where a PU-trained discriminator acts as a "reliability gate." It suppresses rewards in states where a supervised reward model might be delusional, effectively zeroing out rewards in unfamiliar or "hacked" states.
Experimental Results
The approach was evaluated on three tasks: 2D Walker locomotion, pixel-based Lifting, and a complex Stacking task.
Key findings include:
- Superior Performance: On the Stacking task, standard GAIL failed due to overfitting, while nn-PUGAIL achieved near-expert performance with faster convergence.
- Eliminating Reward Hacking: In the Walker task, supervised models often rewarded "fake" walking poses. PURL successfully identified these as unreliable, forcing the agent to learn actual locomotion.
- Robustness to Domain Gaps: When tested in environments with different background distractors, PURL maintained high performance while standard Positive-Negative methods collapsed, proving its resilience to environmental artifacts.
Conclusion
By treating agent data as unlabeled rather than negative, PURL addresses the fundamental issues of reward over- and under-estimation. This framework provides a more robust path for training RL agents in complex, high-dimensional environments without requiring manual reward engineering or privileged information about agent success during training.
Summary updated