Introduction
Transformer models have become the gold standard for Natural Language Processing (NLP). However, they often suffer from a specific efficiency problem known as head domination. In the Multi-Head Attention (MHA) mechanism, the model is designed to use different "heads" to attend to various parts of a sentence simultaneously. In practice, however, a tiny fraction of these heads often perform the majority of the work, while others become redundant or "co-adapted," relying too much on each other rather than learning independent features.
The DropHead Method
To address this, researchers introduced DropHead, a structured dropout technique specifically designed for the Transformer architecture. Unlike standard dropout, which randomly zeroes out individual neurons (a "fine-grained" approach), DropHead shuts down entire attention heads during training.
By removing a whole head, the model is forced to find the same linguistic features—such as syntax or coreference—using different attention heads. This leads to a more "democratic" and robust distribution of labor across the MHA mechanism, preventing the model from becoming over-reliant on a few "super-heads."
Scheduled Dropout Rate
The authors found that applying a constant dropout rate is not optimal. Instead, they introduced a unique V-shaped dropout scheduler. This approach challenges traditional methods like Curriculum Dropout (where dropout increases over time) by emphasizing heavy regularization at the very beginning of training.
The V-shaped schedule operates in three distinct phases:
- Early Stage: High dropout prevents dominant heads from taking over the learning process before other heads have a chance to develop.
- Middle Stage: Low dropout allows the model to "fit" the training data and refine its internal representations.
- Late Stage: High dropout acts as a final regularizer to prevent late-stage overfitting.
Experimental Results
The researchers evaluated Scheduled DropHead on two primary tasks: Machine Translation (WMT 2014 English-German) and a suite of seven text classification benchmarks (including IMDB, Yelp, and SNLI).
Key Findings
- Translation Gains: On the WMT14 En-De task, Scheduled DropHead improved the "Transformer-big" baseline by 0.9 BLEU points, a significant margin in translation research.
- Classification Improvements: The method consistently outperformed standard dropout across all seven classification datasets. For instance, it improved accuracy on the IMDB dataset by approximately 1.1% over a standard Transformer.
- Robustness to Pruning: Models trained with DropHead are much more resilient to pruning. Large percentages of heads can be removed at test time with significantly less impact on accuracy compared to standard models, suggesting the model has learned redundant, high-quality pathways.
- Reduced Head Dominance: Qualitative analysis showed that after removing the "most important" head, models trained with DropHead saw a much smaller performance drop than baseline models.
Limitations and Future Work
While effective, the authors note that finding the exact "dip" and "peak" for the V-shaped schedule requires empirical tuning for different architectures. Future research may explore applying DropHead during the massive pre-training phase of models like BERT or testing its efficacy on complex generative tasks like summarization and dialogue systems.
Summary updated