Learning to Generalize for Sequential Decision Making

Xusen Yin, Ralph M. Weischedel, Jonathan May

Abstract

We consider problems of making sequences of decisions to accomplish tasks, interacting via the medium of language. These problems are often tackled with reinforcement learning approaches. We find that these models do not generalize well when applied to novel task domains. However, the large amount of computation necessary to adequately train and explore the search space of sequential decision making, under a reinforcement learning paradigm, precludes the inclusion of large contextualized language models, which might otherwise enable the desired generalization ability. We introduce a teacher-student imitation learning methodology and a means of converting a reinforcement learning model into a natural language understanding model. Together, these methodologies enable the introduction of contextualized language models into the sequential decision making problem space. We show that models can learn faster and generalize more, leveraging both the imitation learning and the reformulation. Our models exceed teacher performance on various held-out decision problems, by up to 7% on in-domain problems and 24% on out-of-domain problems.

Introduction

Reinforcement learning (RL) has proven effective for mastering complex environments through trial-and-error. However, in language-based sequential decision-making tasks, standard RL agents often struggle with a lack of "commonsense" knowledge. For example, in a text-based adventure game, an agent might attempt semantically nonsensical actions because it treats language as arbitrary tokens rather than meaningful concepts.

While large pre-trained language models (LLMs) like BERT possess the world knowledge necessary to overcome these hurdles, their computational intensity makes direct integration into the RL training loop—which often requires millions of iterations—practically unfeasible. This research addresses this bottleneck by introducing a framework that allows high-capacity language models to learn from more efficient "teacher" agents.

Methodology

The core of the proposed approach is a two-stage Teacher-Student Imitation Learning framework that reformulates reinforcement learning as a supervised natural language understanding (NLU) task.

Comparison between standard DQN training and the proposed teacher-student framework.
  1. Teacher Training (Exploration): A lightweight agent, typically using a Convolutional Neural Network (CNN), is trained via standard Deep Q-Learning (DQN). This agent is computationally efficient, allowing it to perform the extensive exploration needed to map the environment's state space.
  2. Student Training (Generalization): The trained teacher generates a "curriculum pool"—a dataset of game trajectories paired with the teacher's predicted Q-values.
  3. NLU Reformulation: The student model (BERT) is trained to predict these Q-values as a supervised learning task. By concatenating game history and potential actions into a single string, the problem is cast as a regression or classification task, enabling efficient fine-tuning of the BERT model without the need for it to participate in the slow exploration phase.

Experimental Setup

The methodology was tested using TextWorld, a platform for procedurally generating text adventure games. Two primary datasets were used:

  • In-Domain: 4,440 "Cooking" games focused on navigation and recipe execution.
  • Out-of-Domain: 208 "Treasure Hunting" games with distinct goals and mechanics, designed to evaluate the model's ability to generalize its linguistic understanding to novel contexts.

Results and Analysis

The BERT-based student models demonstrated a remarkable ability to generalize beyond the data provided by their teachers.

Performance comparison on out-of-domain games, highlighting the superior generalization of the BERT-NLU-SE student model compared to its teacher.

Key Insights

  • Generalization Performance: The BERT-based student (BERT-NLU-SE) outperformed the teacher by 7% on in-domain tasks and 24% on out-of-domain tasks. This suggests that BERT’s pre-existing semantic knowledge allows it to make better decisions in unfamiliar scenarios than an agent trained from scratch.
  • Training Efficiency: The student model reached peak performance in approximately 500,000 steps, a significant reduction from the 10 million steps required to train the teacher agent.
  • Model Formulation: The researchers found that treating Q-value prediction as a regression task (BERT-NLU-SE) yielded better results than a classification approach (BERT-NLU-CE), likely because regression preserves the relative value differences between various actions.

Conclusion

This paper demonstrates that reformulating sequential decision-making as a supervised NLU task allows for the effective integration of large-scale language models into RL environments. By leveraging a teacher-student framework, models can achieve superior generalization and efficiency. Future research may focus on iterative training loops where the student’s reasoning capabilities are used to further guide environmental exploration.

Summary updated

Abstract

We consider problems of making sequences of decisions to accomplish tasks, interacting via the medium of language. These problems are often tackled with reinforcement learning approaches. We find that these models do not generalize well when applied to novel task domains. However, the large amount of computation necessary to adequately train and explore the search space of sequential decision making, under a reinforcement learning paradigm, precludes the inclusion of large contextualized language models, which might otherwise enable the desired generalization ability. We introduce a teacher-student imitation learning methodology and a means of converting a reinforcement learning model into a natural language understanding model. Together, these methodologies enable the introduction of contextualized language models into the sequential decision making problem space. We show that models can learn faster and generalize more, leveraging both the imitation learning and the reformulation. Our models exceed teacher performance on various held-out decision problems, by up to 7% on in-domain problems and 24% on out-of-domain problems.
BibTeX
@inproceedings{yin2020learning,
  author = {Xusen Yin and Ralph M. Weischedel and Jonathan May},
  title = {{Learning to Generalize for Sequential Decision Making}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.273},
}
Figure