Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformers

Danielle Rothermel, Margaret Li, Tim Rocktäschel, Jakob N. Foerster

TL;DR

This paper reveals that the surprising claim that "frozen" pretrained transformers are superior for cross-modal transfer is actually an artifact of suboptimal hyperparameter tuning. By performing a rigorous learning rate sweep, the authors demonstrate that fully fine-tuned (unfrozen) models consistently outperform frozen ones, reaffirming the importance of proper optimization in machine learning benchmarks.

What Problem Does This Paper Address?

In the quest for "universal computation engines," recent research by Lu et al. (2021) suggested a fascinating shortcut: you could take a large transformer pretrained on text, freeze most of its layers, and achieve state-of-the-art results on entirely different modalities like images or mathematical expressions. This implied that the internal structure learned from language was sufficient to process almost any data type.

However, the authors of this paper noticed a potential red flag: the original study relied on a single, fixed learning rate across different tasks. In deep learning, the "best" model architecture can often be hidden by a "bad" learning rate. This paper investigates whether the purported superiority of Frozen Pretrained Transformers (FPTs) holds up when you actually tune the learning rate for every variant.

Methods

The researchers meticulously recreated the experimental setup of the original study but introduced a critical change: a logarithmic learning rate sweep ranging from $1 \times 10^{-6}$ to $0.01$.

They compared four distinct model configurations:

  1. Frozen Random: Randomly initialized, with only input/output and normalization layers trainable.
  2. Frozen Pretrained: Pretrained on text (GPT-2), with most layers frozen.
  3. Unfrozen Random: Randomly initialized, but all layers are trainable (trained from scratch).
  4. Unfrozen Pretrained: Pretrained on text, and all layers are fine-tuned.

Beyond the learning rate sweep, the authors also implemented proper validation sets for early stopping and fixed significant tokenization bugs in the ListOps dataset that previously made the task unsolvable for long sequences.

Datasets

The study evaluates these models across four diverse tasks representing different modalities:

  • ListOps: Evaluating nested hierarchical mathematical operations.
  • MNIST: Classic handwritten digit classification.
  • CIFAR10: Standard image classification.
  • CIFAR10-LRA: A more challenging "Long Range Arena" version of CIFAR10 where images are processed pixel-by-pixel as a long sequence.

Key Insights and Ideas

  • The "Rug" Effect: The most striking insight is how sensitive the results are to the learning rate. As seen in Figure 1, the learning rate used in previous work ($1 \times 10^{-3}$) sits right at a "cliff" where the performance of unfrozen models collapses. If you only look at that single point, frozen models appear superior. If you look just slightly to the left (lower learning rates), the conclusion completely inverts.
Test accuracy on CIFAR10-LRA showing the performance "cliff" at higher learning rates.
  • Pretraining is Still a Superpower: While freezing layers isn't optimal, the paper confirms that starting from a language-pretrained model provides a significant boost in both final accuracy and convergence speed, even for non-language tasks like pixel-level image classification.
  • Underfitting vs. Overfitting: The authors found that frozen models often fail because they simply don't have enough trainable parameters to fit the new data (underfitting), rather than being "more robust" as previously hypothesized.

Key Results

When the learning rate is properly tuned, the Unfrozen Pretrained variant is the clear winner across the board.

Comparison of test accuracy across all tasks. Unfrozen variants (orange) consistently beat frozen variants (blue).

As summarized in the table below, the performance gaps are often substantial. For the difficult CIFAR10-LRA task, the fully fine-tuned pretrained model achieves 67.8% accuracy, compared to only 54.7% for the frozen version.

Table 1. Comparison of test accuracy across initialization and finetuning methods for the GPT2 architecture.

Furthermore, the authors found that unfrozen models are more computationally efficient. They require fewer gradient steps to reach the maximum performance level that a frozen model is capable of achieving.

Limitations and Future Work

The authors acknowledge they did not replicate every single task from the original paper (omitting the "Bit" tasks as they were too easy and "Homology" due to codebase limitations).

The biggest remaining mystery is why language pretraining helps with images or math at all. While this paper proves that full fine-tuning is the way to go, it leaves open the question of which specific features or "computational structures" are being transferred across these wildly different modalities. Future work could investigate the specific layers or attention heads that contribute most to this cross-modal success.

Summary updated

Abstract

Self-supervised pre-training of large-scale transformer models on text corpora followed by finetuning has achieved state-of-the-art on a number of natural language processing tasks. Recently, Lu et al. (2021, arXiv:2103.05247) claimed that frozen pretrained transformers (FPTs) match or outperform training from scratch as well as unfrozen (fine-tuned) pretrained transformers in a set of transfer tasks to other modalities. In our work, we find that this result is, in fact, an artifact of not tuning the learning rates. After carefully redesigning the empirical setup, we find that when tuning learning rates properly, pretrained transformers do outperform or match training from scratch in all of our tasks, but only as long as the entire model is finetuned. Thus, while transfer from pretrained language models to other modalities does indeed provide gains and hints at exciting possibilities for future work, properly tuning hyperparameters is important for arriving at robust findings.
BibTeX
@misc{rothermel2021don,
  author = {Danielle Rothermel and Margaret Li and Tim Rocktäschel and Jakob N. Foerster},
  title = {{Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformers}},
  year = {2021},
  eprint = {2107.12460},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.12460},
}
Figure