Contrastive introspection (ConSpec) to rapidly identify invariant steps for success

Chen Sun, Wannan Yang, Benjamin Alsbury-Nealy, Yoshua Bengio, Blake A. Richards

Abstract

Contrastive Retrospection (ConSpec) is a plug-and-play reinforcement learning (RL) module designed to solve long-horizon credit assignment by identifying invariant milestones of success. By retrospectively comparing successful and failed episodes, ConSpec learns latent prototypes of critical steps, providing intrinsic rewards that guide agents through sparse-reward environments. Our results demonstrate that ConSpec significantly outperforms traditional RL baselines in complex 3D tasks and notorious benchmarks like Montezuma's Revenge, while exhibiting superior zero-shot generalization to out-of-distribution distractors.

Introduction

Traditional reinforcement learning relies heavily on the Bellman equation for credit assignment, a process that struggles in "long-horizon" tasks where rewards are sparse and separated by long sequences of actions. In such environments, standard agents attempt to propagate value backward step-by-step, which becomes computationally prohibitive when critical events are infrequent.

Humans solve this by using retrospection: looking back at a success to identify the few critical milestones that made it possible while ignoring thousands of irrelevant intermediate states. ConSpec aims to replicate this capability by allowing RL agents to "hone in" on these critical steps through a contrastive learning framework.

Methodology

ConSpec is designed as an add-on architecture that can be paired with any backbone RL algorithm, such as Proximal Policy Optimization (PPO). Its operation is defined by four core components:

  1. Dual Memory Buffers: The system maintains a history of successful episodes ($\mathcal{S}$) and failed episodes ($\mathcal{F}$).
  2. Latent Prototypes: It learns a set of $H$ "prototypes"—vectors in a latent space that represent idealized versions of critical milestones.
  3. Contrastive Loss: This is the system's "secret sauce." The loss function pushes the maximum similarity between a prototype and the states in a successful episode toward 1, while pushing it toward 0 for failed episodes. This forces the model to identify specific features consistently present when winning but missing when losing.
  4. Intrinsic Reward Shaping: Once prototypes are learned, the agent receives an intrinsic bonus reward whenever its current state matches a prototype. This effectively converts a sparse-reward problem into a dense-reward problem guided by meaningful milestones.
The ConSpec architecture integrates with a backbone RL system, using contrastive loss over successful and failed episode buffers to learn latent prototypes.

Unlike previous bottleneck-state methods, ConSpec focuses on learning invariant features (e.g., "holding a flat object") rather than specific coordinates or pixel-perfect matches, allowing for high robustness to environmental distractors.

Evaluation and Results

The authors evaluated ConSpec across diverse environments, including custom 3D Unity tasks (OrangeTree), 2D Gridworlds (Multi-key-to-door), and classic Atari benchmarks.

Solving Complex Long-Horizon Tasks

In the 3D OrangeTree task—which requires searching for a flat object, waiting, and then using that object to reach a goal—ConSpec achieved maximum reward. In contrast, standard PPO and advanced methods like CURL and SynthRs failed completely.

The Orange-Tree task (a) requires multi-stage reasoning. ConSpec significantly outperforms baselines (b) by identifying invariant features of critical states (c), such as the presence of a flat object regardless of color.

Scaling and Efficiency

  • Constant-Time Scaling: In multi-key-to-door tasks, ConSpec's training time remained nearly constant as the number of required keys increased from 2 to 4. Baselines like TVT and RUDDER collapsed as complexity grew.
  • Sample Efficiency: On the notorious "Montezuma’s Revenge," ConSpec made significant progress by learning effectively from as few as two successful trajectories, without requiring dedicated exploration algorithms.
  • Zero-Shot Generalization: Agents trained on colored objects in 3D environments solved tasks immediately when objects were changed to black or backgrounds were altered, demonstrating true out-of-distribution robustness.

Discussion

ConSpec demonstrates that retrospection is often computationally simpler than prospection. By looking back at successful trajectories to find commonalities, agents can bypass the exponential scaling issues of traditional value propagation.

While the current implementation requires a pre-specified number of prototypes ($H$) and a binary definition of success, it represents a significant step toward more human-like credit assignment. Future work may explore dynamic prototype recruitment and the integration of these milestones as sub-goals in Hierarchical Reinforcement Learning (HRL).

Summary updated

Abstract

Reinforcement learning (RL) algorithms have achieved notable success in recent years, but still struggle with fundamental issues in long-term credit assignment. It remains difficult to learn in situations where success is contingent upon multiple critical steps that are distant in time from each other and from a sparse reward; as is often the case in real life. Moreover, how RL algorithms assign credit in these difficult situations is typically not coded in a way that can rapidly generalize to new situations. Here, we present an approach using offline contrastive learning, which we call contrastive introspection (ConSpec), that can be added to any existing RL algorithm and addresses both issues. In ConSpec, a contrastive loss is used during offline replay to identify invariances among successful episodes. This takes advantage of the fact that it is easier to retrospectively identify the small set of steps that success is contingent upon than it is to prospectively predict reward at every step taken in the environment. ConSpec stores this knowledge in a collection of prototypes summarizing the intermediate states required for success. During training, arrival at any state that matches these prototypes generates an intrinsic reward that is added to any external rewards. As well, the reward shaping provided by ConSpec can be made to preserve the optimal policy of the underlying RL agent. The prototypes in ConSpec provide two key benefits for credit assignment: (1) They enable rapid identification of all the critical states. (2) They do so in a readily interpretable manner, enabling out of distribution generalization when sensory features are altered. In summary, ConSpec is a modular system that can be added to any existing RL algorithm to improve its long-term credit assignment.
BibTeX
@misc{sun2022contrastive,
  author = {Chen Sun and Wannan Yang and Benjamin Alsbury-Nealy and Yoshua Bengio and Blake A. Richards},
  title = {{Contrastive introspection (ConSpec) to rapidly identify invariant steps for success}},
  year = {2022},
  url = {https://openreview.net/forum?id=8khKUTiE1Ie},
}
Figure