Can Offline Reinforcement Learning Help Natural Language Understanding?

Ziqi Zhang, Yile Wang, Yue Zhang, Donglin WangarXiv 2022 · OpenReview

Introduction

The Transformer architecture has revolutionized AI, serving as a versatile foundation for tasks across text, vision, and audio. While multi-modal pre-training—combining images and text—is known to enhance linguistic capabilities, the interplay between Reinforcement Learning (RL) and Language Modeling (LM) remains largely unexplored. This research investigates whether the structural patterns learned from predicting sequences in physical environments (trajectories) can transfer to the nuances of human language understanding.

Methodology: From Trajectories to Text

The core hypothesis is that predicting the next "state" in an environment shares fundamental similarities with predicting the next "word" in a sentence, as both rely on capturing local and long-range dependencies.

The transfer learning pipeline: pre-training on RL trajectories using a Decision Transformer and fine-tuning the backbone for NLU tasks.

The researchers employed a two-stage process:

  1. Pre-training (RL Phase): A Decision Transformer (DT) was trained on offline RL datasets. Instead of tokens, the model processed trajectories composed of returns-to-go ($R$), states ($s$), and actions ($a$): $\{R_1, s_1, a_1, ..., R_i, s_i, a_i\}$.
  2. Fine-tuning (NLU Phase): The pre-trained Transformer backbone was extracted and fine-tuned on various Natural Language Understanding (NLU) tasks from the GLUE benchmark.

Experimental Setup

The study utilized the D4RL benchmark for pre-training, covering 11 tasks across robotic control (Gym-Mujoco) and navigation (Maze2d, Antmaze). Evaluation was conducted on six GLUE benchmark tasks, including sentiment analysis (SST2), paraphrase detection (MRPC, QQP), and logical inference (WNLI, QNLI, RTE).

To establish a baseline, the "RL-GPT" model was compared against:

  • LM-GPT: Pre-trained on the Wikitext-103 corpus (standard text pre-training).
  • Random-GPT: A Transformer initialized with random weights.

Key Findings

The results demonstrate that RL pre-training provides a tangible performance boost over random initialization. In 8 out of 11 scenarios, RL-GPT outperformed Random-GPT, achieving an average accuracy of ~65.0 compared to 64.10.

Competitive Performance in Logic

While LM-GPT generally maintained a lead (67.43 average), RL-GPT proved highly competitive and occasionally superior in tasks requiring logical reasoning, such as Recognizing Textual Entailment (RTE) and WNLI. For instance, pre-training on walker2d-medium-replay yielded a nearly 2-point improvement over the random baseline.

The Importance of Sequential Structure

A critical insight is that the model benefits from the temporal structure of RL data rather than just the distribution of values. To prove this, the researchers conducted shuffling experiments.

Shuffling experiments (inter-shuffle) used to test if the model relies on the sequential order of trajectories.

When the order of states and actions was scrambled (inter-shuffle and inner-shuffle), downstream NLU performance significantly declined. This confirms that the "sequential nature" of the data acts as the bridge between the two domains.

Conclusion and Future Work

This research suggests a profound cross-modal synergy: the features required to navigate a robot are transferable to predicting logical entailment in human language. While RL datasets are currently smaller than text corpora, scaling this approach could further bridge the gap. Future work may explore more complex RL environments and investigate whether language pre-training can conversely benefit reinforcement learning agents.

Summary updated

Abstract

Pre-training has been a useful method for learning implicit transferable knowledge and it shows the benefit of offering complementary features across different modalities. Recent work mainly focuses on the modalities such as image and text, for example, studies show that visual features learned from images can help visual-grounded language understanding. In this paper, we consider investigating the potential connection between offline reinforcement learning (RL) and language modeling (LM). Intuitively, RL and LM are similar in predicting the next states based on the current and previous states, which rely on both local and long-range dependency across states. To validate such an assumption, we pre-trained different offline RL tasks using Transformer and then evaluate these models on various language-related tasks. Experimental results show that our RL pre-trained models can give close performance compared with the models using the LM training objective, showing that there exist common useful features across these two modalities. To further explore the potential relationship, we investigate some factors such as Markov property and the sequential nature of RL trajectory.
BibTeX
@misc{zhang2022can,
  author = {Ziqi Zhang and Yile Wang and Yue Zhang and Donglin Wang},
  title = {{Can Offline Reinforcement Learning Help Natural Language Understanding?}},
  year = {2022},
  eprint = {2212.03864},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.03864},
}
Figure