TL;DR
When training transformers on multi-step decision tasks, performance often plateaus for hundreds of epochs before suddenly exploding in a "Eureka-moment." This paper traces this optimization bottleneck to the Softmax function in self-attention, which causes vanishing gradients by allocating uniform attention to incorrect tokens. By introducing simple normalization fixes like NormSoftmax, the authors show how to smooth out the learning process and drastically accelerate convergence.
What Problem Does This Paper Address?
Transformers are the undisputed kings of modern AI, but they occasionally exhibit bizarre training dynamics. One of the most fascinating is the "phase transition"—a sudden, dramatic leap in capability after long periods of stagnation. The authors investigate this phenomenon in the context of multi-step decision tasks.
In these tasks, a model must solve an unsupervised intermediate sub-task (e.g., "where should I look?") to successfully complete the final supervised task (e.g., "classify the object at that location"). While convolutional networks like ResNets learn these multi-step tasks smoothly and immediately, Vision Transformers (ViTs) often get stuck in long performance plateaus, sometimes failing to learn the task entirely depending on the random seed.
Methods
The authors pinpoint the issue to the Softmax function within the transformer's self-attention block. When a transformer focuses on the wrong tokens early in training, the local attention distribution over the relevant tokens becomes uniform. This causes the Jacobian matrix of the Softmax function to collapse toward zero, starving the key ($W_K$) and query ($W_Q$) weight matrices of gradients. Without gradients, the model cannot learn to shift its attention to the correct tokens.
To fix this, the paper evaluates two targeted interventions:
- Heat Treatment (HT): Training begins with a very low Softmax temperature $\tau$ (forcing peakier, low-entropy attention maps) which is gradually increased to the default value ($\sqrt{d_k}$) over time.
- NormSoftmax: The static scaling factor $\sqrt{d_k}$ is replaced with the empirical standard deviation of the query-key dot product, $\sigma(QK^T)$. This dynamically amplifies small differences when attention variance is low, preventing gradient starvation.
Datasets
To isolate the problem from real-world confounding variables, the authors designed several synthetic benchmarks alongside standard real-world datasets:
- Synthetic Vision Tasks: Images composed of four quadrants containing two MNIST digits ("indicators") and two FashionMNIST items ("targets"). The relationship between the indicators dictates which target quadrant to classify. Variants include removing fixed spatial positions ("No position task") and using CIFAR-10 images.
- Reasoning Task: A purely algorithmic symbol-copying task to verify that the problem persists outside of computer vision.
- Real-World Tasks: Masked language modeling with RoBERTa on Wikipedia and In-Context Learning (ICL) with GPT-2 on the Omniglot dataset.
Key Insights and Ideas
- The Information is There, Just Unused: Linear probe analysis revealed that the transformer actually extracts and represents the intermediate "indicator" features early in training. However, it fails to utilize or combine them because the attention mechanism explicitly ignores them.
- Gradient Imbalance: The gradient updates for the value matrix ($W_V$) remain healthy because the model quickly learns to classify the target objects, but the gradients for $W_Q$ and $W_K$ vanish. This creates a flat optimization landscape where the model cannot discover where to look.
- Distinct from Grokking: Unlike "grokking" (where validation accuracy jumps long after training accuracy hits 100% due to overfitting), Eureka-moments feature a simultaneous, sudden drop in both training and validation loss.
Key Results
- Drastic Acceleration: On the main synthetic vision dataset, vanilla ViTs only achieved a Eureka-moment in 7 out of 10 random seeds, taking an average of 207 epochs to break the plateau. Applying NormSoftmax achieved a 10/10 success rate and dropped the average Eureka-epoch down to just 28.
- Robustness to Hyperparameters: NormSoftmax drastically increased the model's robustness to suboptimal learning rates, achieving a perfect success rate across various schedules where vanilla ViTs failed completely.
- Real-World Transfer: The solutions successfully transferred to real-world tasks. NormSoftmax led to significantly earlier loss drops during RoBERTa pretraining on Wikipedia and yielded higher final accuracy in GPT-2 in-context learning on Omniglot.
Limitations and Future Work
While the synthetic tasks perfectly isolate the two-step bottleneck, real-world tasks often involve many more hidden steps, making Eureka-moments harder to predict or track. Additionally, while NormSoftmax and Heat Treatment mitigate Softmax-induced gradient starvation, they do not completely alter the underlying multi-step optimization landscape. The authors suggest that future work should explore more robust architectural changes tailored specifically for complex, hierarchical reasoning.
Summary updated