Abstract: Vincent Micheli, Francois Fleuret. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
2021.
Introduction
In text-based environments like ALFWorld, agents must navigate complex worlds and manipulate objects using only textual descriptions (e.g., "Rinse an egg and put it in the microwave"). This presents two major challenges: an infinite action space where agents must learn a specific command grammar, and extreme data scarcity, as traditional imitation learning requires thousands of time-consuming human demonstrations.
Researchers from the University of Geneva demonstrate that pretrained language models, such as GPT-2, can overcome these hurdles by leveraging the "common sense" semantic knowledge acquired during pretraining. Their agent, GPT-partial*, achieves state-of-the-art performance using only a fraction of the standard expert data.
Methodology: Iterated Action Modeling
The authors propose a two-stage training procedure that treats reinforcement learning as iterative supervised learning:
- Action Modeling (Imitation): A pretrained GPT-2 Medium model is fine-tuned on a minimal set of expert demonstrations (as few as 42). It learns to predict the next action based on the current goal and observation history.
- Iterated Action Modeling (IAM): To improve without further human data, the model explores the environment. Successful trajectories are added back into the training set, allowing the model to "imitate itself" and bootstrap its performance.
Experimental Results
The GPT-based agents significantly outperformed previous specialized models. As shown in the performance comparison, the full GPT agent achieved a 91-95% success rate on ALFWorld tasks, more than doubling the performance of the previous state-of-the-art, BUTLER.
Data Efficiency and Scaling
A key finding is the effectiveness of the IAM loop even with minimal initial data. The "GPT-partial*" model, using only 1.2% of the standard expert demonstrations (42 samples), reached a 60% success rate on unseen tasks—a 51% absolute improvement over existing methods trained on much larger datasets.
Robustness to Natural Language
Because these agents are built on general-purpose language models, they exhibit superior generalization to human-annotated goals. While template-based models struggle with linguistic variations, the GPT agents successfully interpreted natural language instructions (e.g., "warm up apple" vs. "put a hot apple in the fridge"), solving up to 57% of such tasks.
Limitations and Future Directions
While highly effective, the current Iterated Action Modeling approach only learns from successes, ignoring potentially valuable information in failed attempts. Additionally, the 1024-token context window of GPT-2 limits the complexity of tasks the agent can track over long durations. Future research may explore scaling these "butler" instincts to larger models, potentially enabling zero-shot task completion in complex environments.
Summary updated