On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning

Shiro TakagiNeurIPS 2022

TL;DR

This paper investigates why Transformers pre-trained on language (like GPT-2) excel at offline reinforcement learning (RL) tasks, while those pre-trained on images (iGPT) fail catastrophically. The authors discover that language pre-training provides a structural "context-like" prior that allows models to learn efficiently even without historical data, whereas image models suffer from severe optimization hurdles like gradient confusion.

What Problem Does This Paper Address?

Transformers have recently been hailed as "universal computation engines," capable of being fine-tuned for tasks far outside their original domain—including reinforcement learning. Previous research showed a strange phenomenon: pre-training on Wikipedia helps a robot learn to walk in a simulator, but pre-training on ImageNet actually makes it worse than starting from scratch.

The "how" and "why" remained a mystery. Does a language model actually understand the "logic" of a trajectory? Why does a vision model—which arguably deals with spatial data more similar to a physical environment—fail so badly? This paper dives into the "black box" of the Transformer’s internal representations and gradients to find out.

Methods

The authors use the Decision Transformer (DT) architecture as their baseline, comparing three versions:

  1. Randomly Initialized: No pre-training.
  2. GPT-2: Pre-trained on natural language.
  3. iGPT: Pre-trained on image data.

To peek inside the models, they employ several sophisticated analytical tools:

  • CKA (Centered Kernel Alignment): To measure how much internal representations change before and after fine-tuning.
  • MINE (Mutual Information Neural Estimation): To quantify how much information about the RL state/action data is actually being encoded in the hidden layers.
  • Gradient Analysis: Measuring "gradient confusion" (when different data samples suggest contradictory parameter updates) and gradient norms.
  • Ablation Experiments: Specifically, "No-Context" fine-tuning (setting context length $K=1$) and "Block Replacement" (swapping individual blocks of a random model with pre-trained ones).

The study focuses on the D4RL benchmark, a standard for offline RL. They specifically use the "medium" versions of three MuJoCo environments: Hopper, HalfCheetah, and Walker2D.

Key Insights and Ideas

The "Structural Prior" Hypothesis

Language pre-training doesn't teach the model about physics; it teaches the model a superior way to process sequences. The authors found that GPT-2 learns RL tasks efficiently even when historical context is removed ($K=1$). This suggests the pre-trained weights already contain a "context-equivalent" mechanism that helps the model navigate the state space.

iGPT’s Catastrophic Failure is an Optimization Issue

The poor performance of image-pre-trained models isn't just about "bad features." The authors found that iGPT suffers from high gradient confusion and exploding gradients in its Layer Norm parameters. Because the training uses gradient clipping, the useful information in the gradients is essentially "squashed," preventing the model from learning.

Representation Overhaul

Surprisingly, pre-trained models change their internal representations more than randomly initialized ones during fine-tuning, yet they end up encoding less specific information about the downstream data.

CKA similarity across layers for different initializations

As seen in the CKA analysis above, the pre-trained models (GPT-2 and iGPT) exhibit lower similarity between their pre- and post-fine-tuning representations compared to the random initialization, indicating a more significant overhaul of their internal features.

Mutual Information estimation across context

Furthermore, mutual information estimation reveals that the randomly initialized model encodes the most specific information about the downstream data, while GPT-2 encodes less, and iGPT encodes almost none. This suggests successful pre-trained models leverage their pre-existing structural knowledge rather than just memorizing the new dataset.

Middle Layers are the MVP

By replacing single blocks, the authors found that the middle layers of GPT-2 (which typically handle syntax in NLP) are the most beneficial for RL. Replacing just one block in a random model with a pre-trained middle block significantly boosted learning speed.

Key Results

  • Performance Gap: On the Hopper-medium task, the language-pre-trained GPT-2 achieved a normalized score of 79.5, significantly outperforming the Randomly Initialized baseline (67.7). The image-pre-trained iGPT collapsed to a score of 2.6.
  • The Power of No Context: In a "no-context" ($K=1$) setting where the model cannot see past states, GPT-2 still achieved a score of 83.7 on Hopper. In contrast, the Randomly Initialized model failed completely with a score of 0.24. This proves that language pre-training provides a massive head start in understanding how to act, even with minimal information.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Single Random Seed: Due to computational constraints, the study only used one random seed per experiment. In RL, results can vary significantly across seeds, so more extensive testing is needed to ensure robustness.
  • Limited Scope: The study only looked at MuJoCo tasks. It remains to be seen if these insights hold for more complex, high-dimensional RL environments or different Transformer architectures.
  • The "Why" of iGPT Gradients: While they identified that Layer Norm parameters dominate iGPT's gradients, the root cause of why image pre-training leads to this specific optimization failure remains an open question for future research.

Summary updated

BibTeX
@inproceedings{takagi2022effect,
  author = {Shiro Takagi},
  title = {{On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure