TL;DR
This paper explores how "autotelic" AI agents, which learn by setting their own goals, can effectively navigate complex text-based environments. They demonstrate that selective feedback from a "social peer" and prioritizing rare, difficult goals in training are crucial for agents to learn diverse and challenging tasks.
What Problem Does This Paper Address?
The grand challenge in AI is creating agents that can autonomously explore and master a wide range of tasks, much like humans do. This is particularly difficult in environments with vast, text-based action and goal spaces, where traditional reward signals are sparse or non-existent. The authors focus on "autotelic" reinforcement learning (RL) agents, which are intrinsically motivated and learn by defining and pursuing their own goals, effectively creating their own learning curriculum. They investigate how language can aid this process, specifically by leveraging feedback from a social partner and efficiently learning from infrequent but important goal experiences.
Methods
The researchers employ an autotelic RL agent architecture built upon the Deep Reinforcement Relevance Network (DRRN). Key methodological contributions include:
- Social Peer Feedback: Investigating two modes of feedback from a social peer (SP): exhaustive (all possible goals are relabeled) versus selective (only relevant goals are relabeled).
- Experience Replay for Rare Goals: Developing a modular replay buffer that allows for custom sampling strategies to over-sample transitions corresponding to rare, difficult goals.
- Stepping-Stone Exploration: Incorporating mechanisms like "go-explore" (random exploration after task termination) and "goal-chain" (sampling new goals after achieving one) to facilitate learning through a hierarchy of goals.
- Metacognitive Goal Sampling: Introducing a novel approach where the agent samples goals based on its own estimated intermediate competence, aiming to find goals that offer the most learning opportunity.
Datasets
The experiments are conducted within ScienceWorld, a text-based environment designed to test RL agents on complex, procedurally generated household scenarios. ScienceWorld features a rich, combinatorial physics engine and a large, text-based action space, making it a challenging benchmark for goal discovery and exploration. The paper does not introduce a new dataset but uses ScienceWorld to evaluate their proposed methods.
Key Insights and Ideas
- Selective Feedback is Key: An exhaustive social peer providing feedback on every possible goal leads to an overwhelming number of learned goals, drowning out truly informative ones. Selective feedback, focusing on relevant goals, is far more effective for agent learning.
- Don't Forget the Hard Stuff: In environments with varying goal difficulty, standard experience replay buffers tend to be dominated by easy-goal transitions. The agent needs to actively over-sample rare, hard-goal transitions to make progress on challenging tasks.
- Learn from Your Own Progress: Agents that intelligently sample goals based on their current competence (i.e., goals that are challenging but not impossible) learn more effectively and achieve higher overall performance. This "metacognitive" approach allows agents to build their own curriculum.
- Language as a Guide, Not a Crutch: While language is crucial for representing goals and feedback, the quality and selectivity of that language feedback are paramount. Simply providing more language doesn't automatically lead to better learning.
Key Results
The paper presents several configurations of their autotelic agent and evaluates them on ScienceWorld.
- The metacognitive agent, which combines go-explore, goal-chain, and competence-based goal sampling, achieved the highest overall score (87.31 ± 4.97) and the best score on hard goals (76.36 ± 36.52).
- This metacognitive agent also exhibited significantly lower variance across different random seeds compared to other configurations, indicating more stable and reliable learning.
- Configurations with no feedback or unconstrained feedback performed very poorly, highlighting the critical role of appropriate social peer interaction and goal selection.
- The uniform-transition configuration, which sampled goals based on transition frequency, performed worse than the base autotelic agent, underscoring the need for specialized replay mechanisms for rare goals.
Limitations and Future Work
- Environment Specificity: The current work is focused on a single environment (ScienceWorld). Future work could explore generalization across different environment variations or even other text-based games.
- Goal Generation: The current agent samples goals from a predefined set or from achieved goals. A truly open-ended agent should be able to create novel goals based on its prior achievements and understanding.
- Leveraging Large Language Models (LLMs): The authors suggest integrating pre-trained LLMs to enhance the social peer's feedback, allowing for more nuanced and imaginative goal suggestions beyond simple string matching, and potentially leveraging commonsense knowledge.
- Automatic Curriculum Learning: While the metacognitive agent creates a form of self-curriculum, more advanced automatic curriculum learning strategies, such as those based on learning progress, could be explored.
Summary updated