Learning to Generalize for Sequential Decision Making

Xusen Yin, Ralph M. Weischedel, Jonathan May

Introduction

Reinforcement Learning (RL) agents often struggle with "commonsense" reasoning. While humans intuitively understand semantic relationships between objects and actions, a standard RL agent must undergo thousands of failures to learn basic environmental constraints. Large Language Models (LLMs) like BERT possess the world knowledge necessary to bridge this gap; however, their computational requirements make them impractical for the iterative trial-and-error loops of traditional RL.

Researchers from USC’s Information Sciences Institute proposed a teacher-student imitation learning framework to integrate contextualized language models into sequential decision-making agents. This approach allows agents to leverage pre-trained linguistic knowledge without the prohibitive cost of fine-tuning heavy models during exploration.

Methodology: The Teacher-Student Framework

The core innovation lies in decoupling exploration from representation learning through a two-stage process.

1. Teacher Training (RL Phase)

Initially, a lightweight, fast CNN-based agent is trained using a standard Deep Q-Network (DQN) approach. This "teacher" explores the environment over millions of steps to identify rewarding actions.

2. Student Training (Imitation Phase)

Once the teacher achieves competence, it generates a "curriculum pool"—a dataset comprising game states, possible actions, and the teacher's predicted Q-values. The student model then learns to "imitate" these values via supervised learning.

Comparison of normal DQN training and the proposed teacher-student framework.

3. NLU Reformulation

The authors treat the student's learning as a Natural Language Understanding (NLU) task. Instead of encoding states and actions separately, they concatenate the game trajectory and the action into a single string (e.g., [CLS] Trajectory [SEP] Action [SEP]). This allows BERT’s self-attention mechanism to directly model the relationship between the environment and the proposed command, a configuration that significantly outperformed traditional separate encoding.

Experimental Setup

The methodology was evaluated on two distinct datasets:

  • TextWorld (Cooking Genre): 4,440 procedurally generated games used for training and in-domain evaluation.
  • Treasure Hunting (Out-of-Domain): 208 held-out games with different goals and mechanics used to test generalization.

Key Results and Insights

The BERT-based student demonstrated a remarkable ability to generalize beyond its teacher's experience.

  • Out-of-Domain Mastery: The student outperformed the teacher by approximately 24 absolute percentage points on the treasure-hunting tasks. This indicates that the student leveraged its pre-trained linguistic knowledge to navigate novel scenarios that the teacher had never encountered.
  • In-Domain Performance: Even on the training tasks, the BERT student exceeded the teacher's performance by 7%.
  • Computational Efficiency: By converting RL into a supervised NLU task, the researchers utilized larger batches and faster training routines, making the use of BERT feasible.
Performance on out-of-domain Treasure Hunting games, showing the BERT-based student's superior generalization compared to the teacher.

Discussion and Limitations

While successful, the framework is bounded by the teacher's exploration; the student cannot learn from states the teacher never visited. Furthermore, the study focused on limited action spaces, whereas real-world language tasks often involve unbounded possibilities. Future work may explore dynamic interactions where the student can actively query the teacher for labels on high-uncertainty states.

Summary based on: Yin, X., Weischedel, R., & May, J. (2020). Learning to Generalize for Sequential Decision Making. EMNLP 2020.

Summary updated

Abstract

We consider problems of making sequences of decisions to accomplish tasks, interacting via the medium of language. These problems are often tackled with reinforcement learning approaches. We find that these models do not generalize well when applied to novel task domains. However, the large amount of computation necessary to adequately train and explore the search space of sequential decision making, under a reinforcement learning paradigm, precludes the inclusion of large contextualized language models, which might otherwise enable the desired generalization ability. We introduce a teacher-student imitation learning methodology and a means of converting a reinforcement learning model into a natural language understanding model. Together, these methodologies enable the introduction of contextualized language models into the sequential decision making problem space. We show that models can learn faster and generalize more, leveraging both the imitation learning and the reformulation. Our models exceed teacher performance on various held-out decision problems, by up to 7% on in-domain problems and 24% on out-of-domain problems.
BibTeX
@inproceedings{yin2020learning,
  author = {Xusen Yin and Ralph M. Weischedel and Jonathan May},
  title = {{Learning to Generalize for Sequential Decision Making}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.273},
}
Figure