Reinforcement learning (RL) typically relies on manually crafted reward functions, which are notoriously difficult and time-consuming to design. This paper introduces Recursive Classification of Examples (RCE), an algorithm that allows users to specify tasks simply by providing examples of successful outcomes (states). By directly learning a value function from these examples without an intermediate reward-learning step, RCE simplifies the RL pipeline and significantly outperforms prior imitation and reward-learning methods on complex robotic manipulation tasks.
The Problem with Reward Engineering
In standard RL, tasks are defined by a reward function. However, designing a reward function that elicits the desired behavior without unintended side effects—often called "reward hacking"—requires considerable domain expertise.
An alternative is imitation learning, but this requires full expert trajectories, which can be expensive or impossible to collect (e.g., a human might not be able to teleoperate a robot arm effectively). A more user-friendly approach is example-based control, where the user simply provides snapshots of what success looks like, such as an image of a closed door or a properly grasped object.
Prior methods tackle example-based control using a two-stage process:
- Train a classifier to distinguish success examples from random states to act as a learned reward function.
- Run a standard RL algorithm on this learned reward.
This two-stage approach is brittle, introduces potential biases, and requires tuning many hyperparameters. The authors ask: Can we skip the reward function entirely and learn a policy directly from success examples?
Recursive Classification of Examples (RCE)
The authors propose RCE to cut out the middleman. Instead of predicting future rewards, RCE predicts the probability of future success.
RCE trains a classifier to distinguish between "positive" examples (the user-provided success states) and "negative" examples (random transitions collected by the agent). The novel twist is the recursive nature of this classification, which borrows heavily from Temporal Difference (TD) learning.
Instead of just predicting if the current state is a success, the classifier is trained to predict if a success state will be reached in the future. It does this by bootstrapping: the target for the classifier at the current time step is based on the classifier's own prediction at the next time step. Mathematically, the authors prove that this recursive update satisfies a new, data-driven Bellman equation where the standard explicit reward term is entirely replaced by the success examples.
Experimental Setup and Results
The authors evaluated RCE on simulated robotic manipulation environments, providing the agent with just 200 successful outcome states (not full trajectories) per task. The environments included:
- State-based tasks: Five Sawyer manipulation tasks from Meta-World (e.g.,
drawer_open,push,lift,box_close,bin_picking) and two complex tasks (hammer-human-v0,door-human-v0). - Image-based tasks: Three custom tasks requiring the agent to learn directly from pixels.
Superior Performance: RCE significantly outperformed state-of-the-art imitation and reward-learning baselines (AIRL, VICE, DAC, ORIL, PURL) and SQIL across all tested manipulation tasks.
Solving the Unsolvable: On mechanically complex tasks like bin_picking and hammering, RCE successfully learned a competent policy, whereas none of the baseline methods made any progress whatsoever. Furthermore, RCE successfully learned control policies directly from raw image observations. On the challenging sawyer_clear_image task, RCE was the only method that made meaningful progress.
Key Insights and Limitations
- Data-Driven Bellman Equation: The paper mathematically proves that you don't need a reward function to do dynamic programming. RCE satisfies a Bellman equation where the reward is implicitly defined by the probability of a state being in the set of success examples.
- Robustness to User Suboptimality: The authors show that if you assume a worst-case scenario for the user's state distribution, the problem transforms into minimizing an $f$-divergence (specifically, the squared Hellinger distance) between the policy's state occupancy and the success distribution.
- Generalization over Memorization: When trained on image-based tasks where the target object's size is randomized every episode, RCE successfully reaches for novel objects, proving it learns a generalized, functional notion of success rather than just memorizing pixels.
Limitations: While RCE produces highly effective policies, the classifier's actual probability predictions are not well-calibrated, mirroring the well-known issue of Q-value overestimation in standard actor-critic RL. Additionally, both RCE and the baselines struggled to learn the most challenging high-dimensional image-based tasks (like image-based hammering). Integrating explicit representation learning may be an important future step for scaling example-based control to highly complex visual environments.
Summary updated