Abstract
Reinforcement learning (RL) algorithms assume that users specify tasks by manually writing down a reward function. However, this process can be laborious and demands considerable technical expertise. Can we devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes? In this paper, we derive a control algorithm that maximizes the future probability of these successful outcome examples. Prior work has approached similar problems with a two-stage process, first learning a reward function and then optimizing this reward function using another RL algorithm. In contrast, our method directly learns a value function from transitions and successful outcomes, without learning this intermediate reward function. Our method therefore requires fewer hyperparameters to tune and lines of code to debug. We show that our method satisfies a new data-driven Bellman equation, where examples take the place of the typical reward function term. Experiments show that our approach outperforms prior methods that learn explicit reward functions.
Introduction
In traditional reinforcement learning, a human expert must write a mathematical "reward function" to specify the desired behavior for an agent. For instance, to train a robot to open a door, one might reward it for its hand being near the handle, for the handle turning, and for the door swinging open. This "reward engineering" is a major bottleneck: it is laborious, requires deep technical expertise, and often leads to "reward hacking," where the agent finds a loophole to maximize points without actually solving the task.
While prior work attempted to learn from examples (Inverse RL), these methods usually involve a complex two-stage process: first, learning a reward function from the examples, and then running a standard RL algorithm on that learned reward. This adds significant overhead and complexity. This paper introduces Recursive Classification of Examples (RCE), an algorithm that allows users to specify tasks simply by providing examples of successful outcomes, bypassing reward engineering entirely.
Methods: Recursive Classification of Examples
The core innovation of RCE is framing control as a recursive classification problem. Instead of maximizing a scalar reward, the agent learns to classify whether a given state-action pair will lead to a "success state" in the future.
A New Bellman Equation
The authors derive a data-driven version of the Bellman equation. In standard RL, the Bellman equation relates the current value to the immediate reward plus the future value. In RCE, the "reward" term is replaced by a classification loss based on user-provided success examples.
Recursive Classification and End-to-End Learning
The algorithm trains a classifier to distinguish between transitions that lead to success and random transitions. Because "success in the future" depends on "success at the next step," the classifier is trained recursively, analogous to how Q-learning updates value estimates. RCE learns a value function directly from transitions and success outcomes, "cutting out the middleman" of an explicit reward function. This makes the implementation simpler and more robust.
Experimental Evaluation
The authors evaluated RCE on a variety of challenging robotic manipulation tasks:
- Meta-World: A benchmark of Sawyer robot arm tasks including
drawer_open,push,lift,box_close, andbin_picking. - D4RL / Rajeswaran et al.: Complex, high-dimensional tasks like
door-human-v0andhammer-human-v0. - Image-Based Control: Custom tasks where the agent only sees raw pixels, such as
reach_random_position_imageandsawyer_clear_image.
For most tasks, the agent was provided with only 200 static images or state vectors representing a "solved" task (e.g., an image of an open drawer).
Key Results
RCE significantly outperformed state-of-the-art baselines (like VICE, DAC, and SQIL) across almost all tasks. In difficult tasks like hammer-human and bin_picking, RCE was often the only method that made any progress. Notably, RCE demonstrated high efficiency, learning tasks like opening a drawer with as few as one single success example. In image-based tasks, RCE learned a general notion of success, generalizing to objects of different sizes even if those specific sizes were not present in the success examples.
Conclusion and Future Work
RCE demonstrates that RL can be mathematically framed as a binary classification problem ("Will I succeed?") rather than a continuous optimization problem. By using examples, users can specify tasks without needing to write code or understand the underlying mechanics of the solution.
However, limitations remain. The probability estimates produced by RCE are not always perfectly calibrated, leading to occasional overconfidence. Additionally, while RCE works on images, it still struggles with the most complex visual tasks, suggesting that integration with better representation learning is a vital next step. Finally, RCE tells the agent what success looks like but does not necessarily assist in exploration when the agent is far from the goal.
Summary updated