Introduction
The Transformer architecture has revolutionized AI, serving as a versatile foundation for tasks across text, vision, and audio. While multi-modal pre-training—combining images and text—is known to enhance linguistic capabilities, the interplay between Reinforcement Learning (RL) and Language Modeling (LM) remains largely unexplored. This research investigates whether the structural patterns learned from predicting sequences in physical environments (trajectories) can transfer to the nuances of human language understanding.
Methodology: From Trajectories to Text
The core hypothesis is that predicting the next "state" in an environment shares fundamental similarities with predicting the next "word" in a sentence, as both rely on capturing local and long-range dependencies.
The researchers employed a two-stage process:
- Pre-training (RL Phase): A Decision Transformer (DT) was trained on offline RL datasets. Instead of tokens, the model processed trajectories composed of returns-to-go ($R$), states ($s$), and actions ($a$): $\{R_1, s_1, a_1, ..., R_i, s_i, a_i\}$.
- Fine-tuning (NLU Phase): The pre-trained Transformer backbone was extracted and fine-tuned on various Natural Language Understanding (NLU) tasks from the GLUE benchmark.
Experimental Setup
The study utilized the D4RL benchmark for pre-training, covering 11 tasks across robotic control (Gym-Mujoco) and navigation (Maze2d, Antmaze). Evaluation was conducted on six GLUE benchmark tasks, including sentiment analysis (SST2), paraphrase detection (MRPC, QQP), and logical inference (WNLI, QNLI, RTE).
To establish a baseline, the "RL-GPT" model was compared against:
- LM-GPT: Pre-trained on the Wikitext-103 corpus (standard text pre-training).
- Random-GPT: A Transformer initialized with random weights.
Key Findings
The results demonstrate that RL pre-training provides a tangible performance boost over random initialization. In 8 out of 11 scenarios, RL-GPT outperformed Random-GPT, achieving an average accuracy of ~65.0 compared to 64.10.
Competitive Performance in Logic
While LM-GPT generally maintained a lead (67.43 average), RL-GPT proved highly competitive and occasionally superior in tasks requiring logical reasoning, such as Recognizing Textual Entailment (RTE) and WNLI. For instance, pre-training on walker2d-medium-replay yielded a nearly 2-point improvement over the random baseline.
The Importance of Sequential Structure
A critical insight is that the model benefits from the temporal structure of RL data rather than just the distribution of values. To prove this, the researchers conducted shuffling experiments.
When the order of states and actions was scrambled (inter-shuffle and inner-shuffle), downstream NLU performance significantly declined. This confirms that the "sequential nature" of the data acts as the bridge between the two domains.
Conclusion and Future Work
This research suggests a profound cross-modal synergy: the features required to navigate a robot are transferable to predicting logical entailment in human language. While RL datasets are currently smaller than text corpora, scaling this approach could further bridge the gap. Future work may explore more complex RL environments and investigate whether language pre-training can conversely benefit reinforcement learning agents.
Summary updated