Breaking through the learning plateaus of in-context learning in Transformer

Jingwen Fu, Tao Yang, Yuwang Wang, Yan Lü, Nanning Zheng

Abstract

In-context learning, i.e., learning from context examples, is an impressive ability of Transformer. Training Transformers to possess this in-context learning skill is computationally intensive due to the occurrence of learning plateaus, which are periods within the training process where there is minimal or no enhancement in the model's in-context learning capability. To study the mechanism behind the learning plateaus, we conceptually separate a component within the model's internal representation that is exclusively affected by the model's weights. We call this the "weights component", and the remainder is identified as the "context component". By conducting meticulous and controlled experiments on synthetic tasks, we note that the persistence of learning plateaus correlates with compromised functionality of the weights component. Recognizing the impaired performance of the weights component as a fundamental behavior driving learning plateaus, we have developed three strategies to expedite the learning of Transformers. The effectiveness of these strategies is further confirmed in natural language processing tasks. In conclusion, our research demonstrates the feasibility of cultivating a powerful in-context learning ability within AI systems in an eco-friendly manner.

Introduction: The Challenge of Learning Plateaus

In-context learning (ICL) allows a model to adapt to a new task simply by observing a few examples in its prompt. Despite its power, the emergence of this ability during training is often non-linear. Models typically experience a long "learning plateau" where accuracy remains near zero, followed by a rapid transition to high performance. Historically, overcoming these plateaus required massive scaling of model size and compute.

Figure 5: Overview of in-context learning tasks and the characteristic learning plateau where accuracy remains stagnant before a sudden transition.

Deconstructing ICL: Weights vs. Context

To understand why these plateaus occur, the authors propose a conceptual split in the Transformer's internal processing:

  1. The Weights Component: This represents the model's ability to extract features from a single query input using its learned weights.
  2. The Context Component: This represents the model's ability to infer the underlying task rule from the provided context examples.

Using linear probes, the researchers discovered a surprising "weight dysfunction": during the plateau phase, the model's ability to process the query itself actually degrades. The model becomes so overwhelmed by the complexity of the context that it fails to maintain a stable representation of the individual inputs.

Experimental Framework and the 3DShapes Task

The study utilizes a variety of datasets, ranging from simple linear functions to NLP benchmarks like SST-ICL and WordSelection. A primary tool for analysis is the 3DShapes synthetic dataset, which allows for precise control over task complexity.

Figure 6: The 3DShapes experimental setup. The Transformer must identify a latent factor (like color or shape) from context images to predict the label for a prompt image.

In the 3DShapes task, the model is presented with a sequence of images. The "hidden rule" (e.g., "label based on object color") is consistent within a sequence but changes between sequences. The model must use the context to correctly label a final prompt image.

Strategies for Breaking the Plateau

Recognizing that the bottleneck lies in the weights component, the authors developed three interventions to stabilize feature extraction:

  • Weights Warm-up: Pre-training the model on a fixed-rule dataset before introducing the full complexity of random-rule data.
  • Mixed Training: Training on a 50/50 mixture of fixed-rule and random-rule sequences.
  • Extra Loss: Adding an auxiliary loss term that explicitly supervises the model to predict the underlying features of the query, ensuring the weights component remains functional.

Key Results and Efficiency Gains

These interventions successfully eliminated or significantly shortened learning plateaus across various tasks.

  • Compute Efficiency: In the 3DShapes task, a 6.7M parameter model using mixed training reached the same accuracy in 20 epochs that a 13.9M parameter model (twice the size) reached in 50 epochs using standard training.
  • NLP Transfer: The "Extra Loss" strategy accelerated learning on the WordSelection task, demonstrating that these insights apply to natural language processing.
  • Dimensionality: In linear regression tasks, weight dysfunction was found to be a threshold phenomenon, appearing primarily in high-dimensional settings ($d > 30$), where the auxiliary loss proved essential for recovery.

Conclusion

This research provides a mechanical explanation for the sudden emergence of in-context learning. By identifying weight dysfunction as the root cause of learning plateaus, the authors offer practical, "eco-friendly" strategies to train powerful AI systems more efficiently, reducing the need for brute-force scaling.

Summary updated

Abstract

In-context learning, i.e., learning from context examples, is an impressive ability of Transformer. Training Transformers to possess this in-context learning skill is computationally intensive due to the occurrence of learning plateaus, which are periods within the training process where there is minimal or no enhancement in the model's in-context learning capability. To study the mechanism behind the learning plateaus, we conceptually seperate a component within the model's internal representation that is exclusively affected by the model's weights. We call this the "weights component", and the remainder is identified as the "context component". By conducting meticulous and controlled experiments on synthetic tasks, we note that the persistence of learning plateaus correlates with compromised functionality of the weights component. Recognizing the impaired performance of the weights component as a fundamental behavior drives learning plateaus, we have developed three strategies to expedite the learning of Transformers. The effectiveness of these strategies is further confirmed in natural language processing tasks. In conclusion, our research demonstrates the feasibility of cultivating a powerful in-context learning ability within AI systems in an eco-friendly manner.
BibTeX
@inproceedings{fu2024breaking,
  author = {Jingwen Fu and Tao Yang and Yuwang Wang and Yan Lü and Nanning Zheng},
  title = {{Breaking through the learning plateaus of in-context learning in Transformer}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=2K87GFLYWz},
}
Figure