Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey

Cédric Colas, Tristan Karch, Olivier Sigaud, Pierre-Yves Oudeyer

Abstract

This survey explores the frontier of Developmental Reinforcement Learning, providing a formal framework for "autotelic" agents—AI systems that autonomously discover, set, and master their own goals in open-ended environments. By merging developmental robotics with deep reinforcement learning, the authors provide a roadmap for building machines capable of open-ended, lifelong learning.

1. Introduction: The Challenge of Open-Endedness

Traditional Reinforcement Learning (RL) typically relies on "externally motivated" agents. A human engineer defines a specific reward function (e.g., "score points in Mario"), and the agent optimizes for it. However, this approach fails in the face of open-endedness. To create robots that can explore complex environments and learn diverse repertoires of skills—from opening drawers to stacking blocks—we cannot hand-code every possible reward. The "Developmental RL" challenge is to create agents that can:

  1. Represent what a goal is (e.g., an image or a linguistic description).
  2. Generate their own goals to pursue.
  3. Evaluate their own progress without an external "score."

2. The RL-IMGEP Framework

The authors introduce the RL-IMGEP (Reinforcement Learning-based Intrinsically Motivated Goal Exploration Process) framework. This framework extends standard goal-conditioned RL by requiring the agent to solve four distinct learning problems:

  • Goal Embeddings ($z_g$): Learning a compact way to represent goals (e.g., using Variational Autoencoders).
  • Goal Space Support ($Z_G$): Learning which goals are actually possible in the environment.
  • Goal Distribution ($D(z_g)$): Learning a "curriculum"—deciding which goals to practice to maximize learning progress.
  • Goal-Conditioned Reward Function ($R_G$): Learning an internal "critic" that signals success for self-generated goals.

3. Taxonomy of Intrinsically Motivated Processes

The paper categorizes Intrinsic Motivations (IMs) into two primary types: Knowledge-based (driven by surprise or prediction error) and Competence-based (driven by the desire to control the environment and master skills).

Taxonomy of Intrinsically Motivated Processes (IMP) and Goal-Conditioned RL Agents.

As illustrated in Figure 1, the landscape of IMP is diverse. Knowledge-based methods like RND or Disagreement focus on exploration without explicit goals. In contrast, Competence-based IMP (IMGEP) involves internal goal generation. The survey specifically highlights RL-IMGEP (or Autotelic RL), where agents like RIG, CURIOUS, and IMAGINE use reinforcement learning to master self-generated objectives.

4. Key Insights and Mechanisms

The "Autotelic" Shift

The term (from Greek auto for self and telos for goal) emphasizes that the agent is the "end" in itself. It isn't just a tool for a human task; it is an explorer of its own capabilities.

Hindsight Relabeling

A major driver for these agents is Hindsight Relabeling. If an agent tries to stack a block but accidentally knocks it off the table, an autotelic agent reinterprets that trajectory as a successful attempt at the goal "knock the block off the table," learning from every interaction.

Language as a Cognitive Tool

Language provides a compositional goal space. An agent can "imagine" a goal it has never seen (like "a blue dog on a red rug") by combining known linguistic concepts, facilitating creative exploration.

5. Grand Challenges and Future Directions

The authors identify several hurdles for the next generation of autotelic agents:

  1. Time-Extended Goals: Mastering sequences of actions (e.g., "open the door, then go through it") rather than just reaching static states.
  2. Social Learning: Moving from "lone scientists" to "social learners" who can leverage human feedback without losing autonomy.
  3. Creative Imagination: Moving beyond sampling goals from past observations to "imagining" entirely new, out-of-distribution possibilities.

By addressing these challenges, researchers aim to move closer to the versatile, autonomous learning seen in biological intelligence.

Summary updated

BibTeX
@article{colas2022intrinsically,
  author = {Cédric Colas and Tristan Karch and Olivier Sigaud and Pierre-Yves Oudeyer},
  title = {{Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey}},
  journal = {J. Artif. Intell. Res.},
  year = {2022},
  doi = {10.1613/jair.1.13554},
}
Figure