Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration

Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Frier, Peter Ford Dominey, Pierre-Yves Oudeyer

TL;DR

This paper introduces IMAGINE, a novel reinforcement learning architecture that uses language to enable agents to imagine and pursue novel, out-of-distribution goals. This "goal imagination" significantly boosts exploration and generalization in complex environments, mimicking how children learn through language.

What Problem Does This Paper Address?

Building AI agents that can autonomously discover and learn a wide range of skills in open-ended environments is a long-standing challenge. While intrinsically motivated agents can explore and set their own goals, they are often limited to goals within the distribution of known effects. This restricts their ability to make truly creative discoveries. The authors propose that, much like children, AI agents can leverage the compositional nature of language to imagine and target goals they haven't directly experienced, leading to more robust learning and exploration.

Methods

The core of IMAGINE is its "goal imagination" mechanism, inspired by construction grammar. It works by:

  • Language Encoder (Le): An LSTM that maps natural language descriptions of outcomes into goal embeddings.
  • Goal Generator: This module samples target goals from a combination of known goals and imagined goals.
  • Goal Imagination Heuristic (CGH): This is the novel part. It identifies "equivalent words" by looking for sentences with an edit distance of 1 (differing by only one word). These equivalent words can then be substituted to generate new, imagined goals. For example, if the agent knows "grasp red tree" and "grow red tree," it can infer that "grasp" and "grow" are equivalent actions for a "red tree" and imagine "grow red plant" or "grasp blue tree."
  • Modular Architectures: The reward function and policy utilize modular architectures (Deep Sets, gated attention, object-centered representations) that are crucial for generalizing to these imagined goals.
  • Intrinsic Motivation: The agent is driven by an internal reward function that evaluates goal achievement, allowing it to learn without external rewards.
  • Hindsight Experience Replay (HER): Used to efficiently train on both known and imagined goals.

Datasets

  • Playground Environment: A newly introduced procedurally generated 2D environment designed to study goal imagination and combinatorial generalization. It features objects with different types, colors, and sizes, and supports predicates like "go," "grasp," and "grow." The environment is designed to disentangle perception and language understanding from the core problem of goal imagination.
  • Training and Testing Goal Sets (Gtrain, Gtest): The set of achievable goals is partitioned. Gtrain consists of goals described by the social partner (SP), while Gtest contains goals designed to evaluate generalization beyond what SP has explicitly described.

Key Insights and Ideas

  • Language as a Goal Imagination Engine: The paper demonstrates that language isn't just for instruction following but can be a powerful tool for generating novel goals, enabling agents to explore beyond their immediate experience.
  • Compositionality is Key: The ability to combine known concepts (words, predicates, attributes) to form new, plausible goals is what drives creative exploration.
  • Modular Architectures are Essential: The success of goal imagination hinges on modularity in the agent's architecture, allowing it to generalize learned reward functions and policies to unseen goal combinations.
  • Behavioral Adaptation: Agents can learn to refine their behavior based on imagined goals, even if their initial zero-shot generalization is imperfect (e.g., learning that plants only need water, not food).

Key Results

  • Significant Generalization Boost: IMAGINE agents with goal imagination achieved significantly higher success rates on unseen testing goals (SRtest) compared to agents without imagination, especially when imagination started early. For example, SRtest improved from ~0.15 to ~0.76 with imagination.
  • Improved Exploration: The "Interesting Interaction Count" (I2C) metric showed that agents with goal imagination explored more diverse and meaningful interactions, particularly on testing and "extra" interaction sets.
  • Modular Architectures Outperform Flat Ones: Modular architectures were crucial for achieving high generalization and enabling the benefits of goal imagination. Flat architectures showed poor generalization and did not benefit from imagination.
  • Robustness to Social Feedback: The system showed resilience to variations in social partner feedback, performing well even with less frequent descriptions.

Limitations and Future Work

  • Limited Language Complexity: The current grammar is relatively simple. Future work could explore more complex language, including object relationships.
  • Pre-trained Language Models: The current approach does not use pre-trained language models, adhering to a developmental approach. Investigating how pre-trained models interact with goal imagination is a future direction.
  • Unreliable Feedback: The paper assumes a mostly reliable social partner. Relaxing this assumption (lower precision) and exploring more realistic feedback scenarios is a future avenue.
  • Hierarchical Settings: Applying goal imagination to hierarchical reinforcement learning for low-level exploration is suggested.
  • Filtering Imagined Goals: While the current mechanism generates imagined goals, future work could focus on more sophisticated methods for filtering meaningless or unachievable goals.
  • Goal-as-State vs. Abstract Goals: The authors argue that goal-as-state approaches are not directly comparable due to the abstract nature of language-conditioned goals.

Summary updated

BibTeX
@inproceedings{colas2020language,
  author = {Cédric Colas and Tristan Karch and Nicolas Lair and Jean-Michel Dussoux and Clément Moulin-Frier and Peter Ford Dominey and Pierre-Yves Oudeyer},
  title = {{Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=Hnv89Oa4L4x},
}
Figure