TL;DR
This paper introduces "DropHead," a novel regularization technique for Transformer models that drops entire attention heads during training. This encourages all attention heads to contribute more, preventing a few dominant heads from taking over. They further propose a "Scheduled DropHead" that dynamically adjusts the dropout rate throughout training, leading to improved performance on machine translation and text classification tasks.
What Problem Does This Paper Address?
The Transformer architecture, a cornerstone of modern NLP, relies heavily on the Multi-Head Attention (MHA) mechanism. While MHA is designed to allow models to attend to information from different representation subspaces, recent research has shown that in practice, a small subset of attention heads often dominates the MHA's output. This "head domination" phenomenon means that many attention heads are underutilized, potentially leading to overfitting and hindering the model's overall generalization ability. The authors aim to address this by developing a regularization method that forces all attention heads to be more robust and contribute meaningfully.
Methods
The core innovation is DropHead, a structured dropout method. Unlike standard dropout, which randomly drops individual units or connections within an attention head, DropHead randomly drops entire attention heads during training with a predefined probability. This forces the remaining heads to compensate and learn more effectively.
To further enhance regularization, the authors propose Scheduled DropHead. They observe that head domination tends to occur early in training. Their proposed dropout rate scheduler adopts a V-shaped curve: it starts with a relatively high dropout rate, gradually decreases it to zero during the initial training phase (often coinciding with learning rate warmup), and then linearly increases it again towards the end of training. This strategy aims to prevent early head domination and then allow the model to learn more effectively in later stages.
Datasets
The paper evaluates their methods on several benchmark datasets:
- Machine Translation: WMT 2014 English-German dataset.
- Text Classification:
- Sentiment Analysis: IMDB, Yelp Review
- Topic Classification: AG's News, DBPedia
- Question Classification: TREC, Yahoo! Answers
- Inference: SNLI
Key Insights and Ideas
- Attention Heads Aren't Equal: The paper confirms and builds upon the finding that not all attention heads in a Transformer are equally important, with some dominating the learning process.
- Dropping Heads Forces Robustness: By dropping entire attention heads, DropHead compels the model to rely on a diverse set of heads, leading to better generalization and preventing overfitting.
- Training Dynamics Matter for Regularization: The V-shaped dropout schedule is motivated by the observation that head domination is an early-stage problem, and a dynamic schedule can be more effective than a constant dropout rate.
- DropHead is More Than Just Regularization: The analysis suggests that DropHead not only regularizes but also makes the MHA mechanism more robust to pruning, meaning attention heads can be removed at inference time with less performance degradation.
Key Results
- Machine Translation: Scheduled DropHead improved the WMT14 En-De translation task by 0.9 BLEU score compared to the baseline Transformer model.
- Text Classification: Achieved around 1.0 accuracy improvement on various text classification tasks.
- Pruning Robustness: Models trained with Scheduled DropHead showed significantly less performance degradation when attention heads were pruned, indicating improved robustness.
- Ablation Studies:
- Scheduled DropHead outperformed vanilla dropout applied to self-attention layers.
- The V-shaped dropout schedule was more effective than constant, curriculum, or anti-curriculum schedules.
Limitations and Future Work
- BERT Pretraining: The authors note that applying DropHead to pre-trained models like BERT might be less impactful because dominant attention heads might already be formed during pretraining. They suggest pretraining BERT with DropHead as a future direction.
- Hyperparameter Tuning: While they aim to minimize additional hyperparameters, the optimal dropout rate and schedule parameters might still require tuning for specific tasks.
- Future Work: The authors plan to extend DropHead to pre-training language models (like BERT) and apply it to other NLP tasks such as text summarization and dialogue systems.
Summary updated