Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems

David T. Hoffmann, Simon Schrodi, Jelena Bratulić, Nadine Behrmann, Volker Fischer, Thomas Brox

TL;DR

When training transformers on multi-step decision tasks, performance often plateaus for hundreds of epochs before suddenly exploding in a "Eureka-moment." This paper traces this optimization bottleneck to the Softmax function in self-attention, which causes vanishing gradients by allocating uniform attention to incorrect tokens. By introducing simple normalization fixes like NormSoftmax, the authors show how to smooth out the learning process and drastically accelerate convergence.

What Problem Does This Paper Address?

Transformers are the undisputed kings of modern AI, but they occasionally exhibit bizarre training dynamics. One of the most fascinating is the "phase transition"—a sudden, dramatic leap in capability after long periods of stagnation. The authors investigate this phenomenon in the context of multi-step decision tasks.

In these tasks, a model must solve an unsupervised intermediate sub-task (e.g., "where should I look?") to successfully complete the final supervised task (e.g., "classify the object at that location"). While convolutional networks like ResNets learn these multi-step tasks smoothly and immediately, Vision Transformers (ViTs) often get stuck in long performance plateaus, sometimes failing to learn the task entirely depending on the random seed.

Figure 1: Transformers can get stuck during optimization for two-stage tasks. (a) Describes our 2 step decision task used to study Eureka-moments. Task 1 is to compare the two indicators (here digits). If the digits are the same, task 2 is to classify the top-right image and bottom left else. Top-right and bottom left are referred to as targets. The location of the correct target is referred to as target location. (b) Validation accuracy and training loss for the task in (a). 2 ViTs (blue and green) fail to converge, while one ViT (yellow) has a Eureka-moment. Eureka-moments are characterised by a sudden increase of accuracy and drop of the loss (in contrast to Grokking (Power et al., 2022)). ResNets are not susceptible to this kind of optimization difficulty. (c) Eureka-moments on real datasets. Sharp improvements after initial plateauing can also be observed for GPT-2 ICL, here in the Omniglot ICL task (Chan et al., 2022) and language modeling with RoBERTa on Wikipedia. We will show later that our analysis transfers to these tasks (see Fig. 9).

Methods

The authors pinpoint the issue to the Softmax function within the transformer's self-attention block. When a transformer focuses on the wrong tokens early in training, the local attention distribution over the relevant tokens becomes uniform. This causes the Jacobian matrix of the Softmax function to collapse toward zero, starving the key ($W_K$) and query ($W_Q$) weight matrices of gradients. Without gradients, the model cannot learn to shift its attention to the correct tokens.

To fix this, the paper evaluates two targeted interventions:

  1. Heat Treatment (HT): Training begins with a very low Softmax temperature $\tau$ (forcing peakier, low-entropy attention maps) which is gradually increased to the default value ($\sqrt{d_k}$) over time.
  2. NormSoftmax: The static scaling factor $\sqrt{d_k}$ is replaced with the empirical standard deviation of the query-key dot product, $\sigma(QK^T)$. This dynamically amplifies small differences when attention variance is low, preventing gradient starvation.

Datasets

To isolate the problem from real-world confounding variables, the authors designed several synthetic benchmarks alongside standard real-world datasets:

  • Synthetic Vision Tasks: Images composed of four quadrants containing two MNIST digits ("indicators") and two FashionMNIST items ("targets"). The relationship between the indicators dictates which target quadrant to classify. Variants include removing fixed spatial positions ("No position task") and using CIFAR-10 images.
  • Reasoning Task: A purely algorithmic symbol-copying task to verify that the problem persists outside of computer vision.
  • Real-World Tasks: Masked language modeling with RoBERTa on Wikipedia and In-Context Learning (ICL) with GPT-2 on the Omniglot dataset.

Key Insights and Ideas

  • The Information is There, Just Unused: Linear probe analysis revealed that the transformer actually extracts and represents the intermediate "indicator" features early in training. However, it fails to utilize or combine them because the attention mechanism explicitly ignores them.
  • Gradient Imbalance: The gradient updates for the value matrix ($W_V$) remain healthy because the model quickly learns to classify the target objects, but the gradients for $W_Q$ and $W_K$ vanish. This creates a flat optimization landscape where the model cannot discover where to look.
  • Distinct from Grokking: Unlike "grokking" (where validation accuracy jumps long after training accuracy hits 100% due to overfitting), Eureka-moments feature a simultaneous, sudden drop in both training and validation loss.

Key Results

  • Drastic Acceleration: On the main synthetic vision dataset, vanilla ViTs only achieved a Eureka-moment in 7 out of 10 random seeds, taking an average of 207 epochs to break the plateau. Applying NormSoftmax achieved a 10/10 success rate and dropped the average Eureka-epoch down to just 28.
  • Robustness to Hyperparameters: NormSoftmax drastically increased the model's robustness to suboptimal learning rates, achieving a perfect success rate across various schedules where vanilla ViTs failed completely.
  • Real-World Transfer: The solutions successfully transferred to real-world tasks. NormSoftmax led to significantly earlier loss drops during RoBERTa pretraining on Wikipedia and yielded higher final accuracy in GPT-2 in-context learning on Omniglot.

Limitations and Future Work

While the synthetic tasks perfectly isolate the two-step bottleneck, real-world tasks often involve many more hidden steps, making Eureka-moments harder to predict or track. Additionally, while NormSoftmax and Heat Treatment mitigate Softmax-induced gradient starvation, they do not completely alter the underlying multi-step optimization landscape. The authors suggest that future work should explore more robust architectural changes tailored specifically for complex, hierarchical reasoning.

Summary updated

Abstract

In this work, we study rapid improvements of the training loss in transformers when being confronted with multi-step decision tasks. We found that transformers struggle to learn the intermediate task and both training and validation loss saturate for hundreds of epochs. When transformers finally learn the intermediate task, they do this rapidly and unexpectedly. We call these abrupt improvements Eureka-moments, since the transformer appears to suddenly learn a previously incomprehensible concept. We designed synthetic tasks to study the problem in detail, but the leaps in performance can be observed also for language modeling and in-context learning (ICL). We suspect that these abrupt transitions are caused by the multi-step nature of these tasks. Indeed, we find connections and show that ways to improve on the synthetic multi-step tasks can be used to improve the training of language modeling and ICL. Using the synthetic data we trace the problem back to the Softmax function in the self-attention block of transformers and show ways to alleviate the problem. These fixes reduce the required number of training steps, lead to higher likelihood to learn the intermediate task, to higher final accuracy and training becomes more robust to hyper-parameters.
BibTeX
@inproceedings{hoffmann2024eureka,
  author = {David T. Hoffmann and Simon Schrodi and Jelena Bratulić and Nadine Behrmann and Volker Fischer and Thomas Brox},
  title = {{Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=HssOwuZiaB},
}
Figure