Abstract
Contrastive Retrospection (ConSpec) is a plug-and-play reinforcement learning (RL) module designed to solve long-horizon credit assignment by identifying invariant milestones of success. By retrospectively comparing successful and failed episodes, ConSpec learns latent prototypes of critical steps, providing intrinsic rewards that guide agents through sparse-reward environments. Our results demonstrate that ConSpec significantly outperforms traditional RL baselines in complex 3D tasks and notorious benchmarks like Montezuma's Revenge, while exhibiting superior zero-shot generalization to out-of-distribution distractors.
Introduction
Traditional reinforcement learning relies heavily on the Bellman equation for credit assignment, a process that struggles in "long-horizon" tasks where rewards are sparse and separated by long sequences of actions. In such environments, standard agents attempt to propagate value backward step-by-step, which becomes computationally prohibitive when critical events are infrequent.
Humans solve this by using retrospection: looking back at a success to identify the few critical milestones that made it possible while ignoring thousands of irrelevant intermediate states. ConSpec aims to replicate this capability by allowing RL agents to "hone in" on these critical steps through a contrastive learning framework.
Methodology
ConSpec is designed as an add-on architecture that can be paired with any backbone RL algorithm, such as Proximal Policy Optimization (PPO). Its operation is defined by four core components:
- Dual Memory Buffers: The system maintains a history of successful episodes ($\mathcal{S}$) and failed episodes ($\mathcal{F}$).
- Latent Prototypes: It learns a set of $H$ "prototypes"—vectors in a latent space that represent idealized versions of critical milestones.
- Contrastive Loss: This is the system's "secret sauce." The loss function pushes the maximum similarity between a prototype and the states in a successful episode toward 1, while pushing it toward 0 for failed episodes. This forces the model to identify specific features consistently present when winning but missing when losing.
- Intrinsic Reward Shaping: Once prototypes are learned, the agent receives an intrinsic bonus reward whenever its current state matches a prototype. This effectively converts a sparse-reward problem into a dense-reward problem guided by meaningful milestones.
Unlike previous bottleneck-state methods, ConSpec focuses on learning invariant features (e.g., "holding a flat object") rather than specific coordinates or pixel-perfect matches, allowing for high robustness to environmental distractors.
Evaluation and Results
The authors evaluated ConSpec across diverse environments, including custom 3D Unity tasks (OrangeTree), 2D Gridworlds (Multi-key-to-door), and classic Atari benchmarks.
Solving Complex Long-Horizon Tasks
In the 3D OrangeTree task—which requires searching for a flat object, waiting, and then using that object to reach a goal—ConSpec achieved maximum reward. In contrast, standard PPO and advanced methods like CURL and SynthRs failed completely.
Scaling and Efficiency
- Constant-Time Scaling: In multi-key-to-door tasks, ConSpec's training time remained nearly constant as the number of required keys increased from 2 to 4. Baselines like TVT and RUDDER collapsed as complexity grew.
- Sample Efficiency: On the notorious "Montezuma’s Revenge," ConSpec made significant progress by learning effectively from as few as two successful trajectories, without requiring dedicated exploration algorithms.
- Zero-Shot Generalization: Agents trained on colored objects in 3D environments solved tasks immediately when objects were changed to black or backgrounds were altered, demonstrating true out-of-distribution robustness.
Discussion
ConSpec demonstrates that retrospection is often computationally simpler than prospection. By looking back at successful trajectories to find commonalities, agents can bypass the exponential scaling issues of traditional value propagation.
While the current implementation requires a pre-specified number of prototypes ($H$) and a binary definition of success, it represents a significant step toward more human-like credit assignment. Future work may explore dynamic prototype recruitment and the integration of these milestones as sub-goals in Hierarchical Reinforcement Learning (HRL).
Summary updated