Introduction
Reinforcement Learning (RL) agents often struggle with "commonsense" reasoning. While humans intuitively understand semantic relationships between objects and actions, a standard RL agent must undergo thousands of failures to learn basic environmental constraints. Large Language Models (LLMs) like BERT possess the world knowledge necessary to bridge this gap; however, their computational requirements make them impractical for the iterative trial-and-error loops of traditional RL.
Researchers from USC’s Information Sciences Institute proposed a teacher-student imitation learning framework to integrate contextualized language models into sequential decision-making agents. This approach allows agents to leverage pre-trained linguistic knowledge without the prohibitive cost of fine-tuning heavy models during exploration.
Methodology: The Teacher-Student Framework
The core innovation lies in decoupling exploration from representation learning through a two-stage process.
1. Teacher Training (RL Phase)
Initially, a lightweight, fast CNN-based agent is trained using a standard Deep Q-Network (DQN) approach. This "teacher" explores the environment over millions of steps to identify rewarding actions.
2. Student Training (Imitation Phase)
Once the teacher achieves competence, it generates a "curriculum pool"—a dataset comprising game states, possible actions, and the teacher's predicted Q-values. The student model then learns to "imitate" these values via supervised learning.
3. NLU Reformulation
The authors treat the student's learning as a Natural Language Understanding (NLU) task. Instead of encoding states and actions separately, they concatenate the game trajectory and the action into a single string (e.g., [CLS] Trajectory [SEP] Action [SEP]). This allows BERT’s self-attention mechanism to directly model the relationship between the environment and the proposed command, a configuration that significantly outperformed traditional separate encoding.
Experimental Setup
The methodology was evaluated on two distinct datasets:
- TextWorld (Cooking Genre): 4,440 procedurally generated games used for training and in-domain evaluation.
- Treasure Hunting (Out-of-Domain): 208 held-out games with different goals and mechanics used to test generalization.
Key Results and Insights
The BERT-based student demonstrated a remarkable ability to generalize beyond its teacher's experience.
- Out-of-Domain Mastery: The student outperformed the teacher by approximately 24 absolute percentage points on the treasure-hunting tasks. This indicates that the student leveraged its pre-trained linguistic knowledge to navigate novel scenarios that the teacher had never encountered.
- In-Domain Performance: Even on the training tasks, the BERT student exceeded the teacher's performance by 7%.
- Computational Efficiency: By converting RL into a supervised NLU task, the researchers utilized larger batches and faster training routines, making the use of BERT feasible.
Discussion and Limitations
While successful, the framework is bounded by the teacher's exploration; the student cannot learn from states the teacher never visited. Furthermore, the study focused on limited action spaces, whereas real-world language tasks often involve unbounded possibilities. Future work may explore dynamic interactions where the student can actively query the teacher for labels on high-uncertainty states.
Summary based on: Yin, X., Weischedel, R., & May, J. (2020). Learning to Generalize for Sequential Decision Making. EMNLP 2020.
Summary updated