TL;DR
This paper reveals that the surprising claim that "frozen" pretrained transformers are superior for cross-modal transfer is actually an artifact of suboptimal hyperparameter tuning. By performing a rigorous learning rate sweep, the authors demonstrate that fully fine-tuned (unfrozen) models consistently outperform frozen ones, reaffirming the importance of proper optimization in machine learning benchmarks.
What Problem Does This Paper Address?
In the quest for "universal computation engines," recent research by Lu et al. (2021) suggested a fascinating shortcut: you could take a large transformer pretrained on text, freeze most of its layers, and achieve state-of-the-art results on entirely different modalities like images or mathematical expressions. This implied that the internal structure learned from language was sufficient to process almost any data type.
However, the authors of this paper noticed a potential red flag: the original study relied on a single, fixed learning rate across different tasks. In deep learning, the "best" model architecture can often be hidden by a "bad" learning rate. This paper investigates whether the purported superiority of Frozen Pretrained Transformers (FPTs) holds up when you actually tune the learning rate for every variant.
Methods
The researchers meticulously recreated the experimental setup of the original study but introduced a critical change: a logarithmic learning rate sweep ranging from $1 \times 10^{-6}$ to $0.01$.
They compared four distinct model configurations:
- Frozen Random: Randomly initialized, with only input/output and normalization layers trainable.
- Frozen Pretrained: Pretrained on text (GPT-2), with most layers frozen.
- Unfrozen Random: Randomly initialized, but all layers are trainable (trained from scratch).
- Unfrozen Pretrained: Pretrained on text, and all layers are fine-tuned.
Beyond the learning rate sweep, the authors also implemented proper validation sets for early stopping and fixed significant tokenization bugs in the ListOps dataset that previously made the task unsolvable for long sequences.
Datasets
The study evaluates these models across four diverse tasks representing different modalities:
- ListOps: Evaluating nested hierarchical mathematical operations.
- MNIST: Classic handwritten digit classification.
- CIFAR10: Standard image classification.
- CIFAR10-LRA: A more challenging "Long Range Arena" version of CIFAR10 where images are processed pixel-by-pixel as a long sequence.
Key Insights and Ideas
- The "Rug" Effect: The most striking insight is how sensitive the results are to the learning rate. As seen in Figure 1, the learning rate used in previous work ($1 \times 10^{-3}$) sits right at a "cliff" where the performance of unfrozen models collapses. If you only look at that single point, frozen models appear superior. If you look just slightly to the left (lower learning rates), the conclusion completely inverts.
- Pretraining is Still a Superpower: While freezing layers isn't optimal, the paper confirms that starting from a language-pretrained model provides a significant boost in both final accuracy and convergence speed, even for non-language tasks like pixel-level image classification.
- Underfitting vs. Overfitting: The authors found that frozen models often fail because they simply don't have enough trainable parameters to fit the new data (underfitting), rather than being "more robust" as previously hypothesized.
Key Results
When the learning rate is properly tuned, the Unfrozen Pretrained variant is the clear winner across the board.
As summarized in the table below, the performance gaps are often substantial. For the difficult CIFAR10-LRA task, the fully fine-tuned pretrained model achieves 67.8% accuracy, compared to only 54.7% for the frozen version.
Furthermore, the authors found that unfrozen models are more computationally efficient. They require fewer gradient steps to reach the maximum performance level that a frozen model is capable of achieving.
Limitations and Future Work
The authors acknowledge they did not replicate every single task from the original paper (omitting the "Bit" tasks as they were too easy and "Homology" due to codebase limitations).
The biggest remaining mystery is why language pretraining helps with images or math at all. While this paper proves that full fine-tuning is the way to go, it leaves open the question of which specific features or "computational structures" are being transferred across these wildly different modalities. Future work could investigate the specific layers or attention heads that contribute most to this cross-modal success.
Summary updated