Think Before You Act: Unified Policy for Interleaving Language Reasoning with Actions

Lina Mezghani, Piotr Bojanowski, Karteek Alahari, Sainbayar Sukhbaatar

Abstract

The success of transformer models trained with a language modeling objective brings a promising opportunity to the reinforcement learning framework. Decision Transformer is a step towards this direction, showing how to train transformers with a similar next-step prediction objective on offline data. Another important development in this area is the recent emergence of large-scale datasets collected from the internet, such as the ones composed of tutorial videos with captions where people talk about what they are doing. To take advantage of this language component, we propose a novel method for unifying language reasoning with actions in a single policy. Specifically, we augment a transformer policy with word outputs, so it can generate textual captions interleaved with actions. When tested on the most challenging task in BabyAI, with captions describing next subgoals, our reasoning policy consistently outperforms the caption-free baseline.

Introduction

Reinforcement Learning (RL) agents often struggle with tasks that require long-term planning and complex sequential reasoning. While Large Language Models (LLMs) have demonstrated an exceptional capacity for "chain-of-thought" reasoning, bridging the gap between linguistic logic and embodied action remains a significant challenge in the field.

Current offline RL approaches, such as the Decision Transformer, typically focus on mapping states directly to actions. However, this ignores a wealth of "reasoning" data available in large-scale human-generated datasets—like YouTube tutorials—where experts explain their intent and subgoals while performing tasks. This paper addresses the challenge of creating an architecture that can seamlessly interleave "thinking" (generating text) and "acting" (generating environment commands) within a single, unified policy.

Methodology: The Unified Policy

The proposed Unified Policy is based on an autoregressive transformer architecture (a reduced GPT-2). The model treats actions and natural language tokens as part of the same sequence, mapped to a unified embedding space.

The unified transformer architecture interleaves language instructions, observations, captions (reasoning), and actions into a single sequence.

Interleaved Sequence Modeling

To synchronize reasoning with action, the authors developed a method to interleave these sequences. Because a single "thought" or caption might correspond to multiple actions—or vice versa—the model duplicates observations when a caption is present. As shown in the architecture diagram, for a given observation $o_t$, the model first predicts a caption $c_t$ (representing a subgoal like "explore" or "goto box") and then uses the same state context to predict the subsequent action $a_t$.

Causal Reasoning and Execution

The model is trained to minimize cross-entropy loss across the entire sequence of instructions, observations, captions, and actions. At inference time, the model's output determines its next step:

  • If it generates a caption token, it feeds the token back into itself to continue the reasoning chain.
  • If it generates an action token, it executes the command in the environment.

Experimental Setup

The study utilizes BabyAI, a grid-world platform designed for language-grounded RL. The evaluation focuses on three levels of increasing difficulty:

  1. PutNextLocal: Basic object manipulation.
  2. Synth: Complex instruction following (e.g., "go to the red ball").
  3. BossLevel: The most challenging task, requiring multiple sequential sub-tasks.

Since standard BabyAI lacks reasoning text, the authors used an internal expert bot to generate a dataset of trajectories augmented with natural language subgoals, such as "explore," "go to yellow box," and "pickup."

Results and Key Insights

The experimental results demonstrate that language reasoning serves as a powerful "divide and conquer" strategy. By predicting a subgoal first, the model effectively constrains its search space for subsequent actions, breaking down complex, long-horizon tasks into manageable chunks.

Performance Gains

On the most difficult BossLevel tasks, the unified policy achieved an 85.2% success rate, significantly outperforming the caption-free baseline's 73.6%. The benefits of "thinking" were most pronounced as task complexity increased; on simpler tasks, the reasoning model performed similarly to standard baselines, but the performance gap widened substantially on long-horizon challenges.

Generalization and Efficiency

The model showed robust zero-shot generalization to unseen maze configurations. Furthermore, the reasoning-augmented model proved more sample-efficient, often outperforming baselines trained on significantly larger datasets. This suggests that language provides a more structured and efficient way to learn complex behaviors than pure state-action mapping.

Conclusion

This work demonstrates that unifying language reasoning and action into a single autoregressive sequence significantly improves an agent's ability to handle complex tasks. While currently tested on synthetic subgoals in a controlled grid-world, the success of this approach paves the way for scaling to high-dimensional environments like Minecraft, where diverse, real-world human captions can provide the necessary common-sense reasoning for expert-level performance.

Summary updated

Abstract

The success of transformer models trained with a language modeling objective brings a promising opportunity to the reinforcement learning framework. Decision Transformer is a step towards this direction, showing how to train transformers with a similar next-step prediction objective on offline data. Another important development in this area is the recent emergence of large-scale datasets collected from the internet, such as the ones composed of tutorial videos with captions where people talk about what they are doing. To take advantage of this language component, we propose a novel method for unifying language reasoning with actions in a single policy. Specifically, we augment a transformer policy with word outputs, so it can generate textual captions interleaved with actions. When tested on the most challenging task in BabyAI, with captions describing next subgoals, our reasoning policy consistently outperforms the caption-free baseline.
BibTeX
@inproceedings{mezghani2023think,
  author = {Lina Mezghani and Piotr Bojanowski and Karteek Alahari and Sainbayar Sukhbaatar},
  title = {{Think Before You Act: Unified Policy for Interleaving Language Reasoning with Actions}},
  booktitle = {RRL @ ICLR},
  year = {2023},
  url = {https://openreview.net/forum?id=HQmhQIi6mN},
}
Figure