Scheduled DropHead: A Regularization Method for Transformer Models

Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, Ke Xu

Introduction

Transformer models have become the gold standard for Natural Language Processing (NLP). However, they often suffer from a specific efficiency problem known as head domination. In the Multi-Head Attention (MHA) mechanism, the model is designed to use different "heads" to attend to various parts of a sentence simultaneously. In practice, however, a tiny fraction of these heads often perform the majority of the work, while others become redundant or "co-adapted," relying too much on each other rather than learning independent features.

The DropHead Method

To address this, researchers introduced DropHead, a structured dropout technique specifically designed for the Transformer architecture. Unlike standard dropout, which randomly zeroes out individual neurons (a "fine-grained" approach), DropHead shuts down entire attention heads during training.

Comparison between (a) Standard Dropout and (b) DropHead mechanisms.

By removing a whole head, the model is forced to find the same linguistic features—such as syntax or coreference—using different attention heads. This leads to a more "democratic" and robust distribution of labor across the MHA mechanism, preventing the model from becoming over-reliant on a few "super-heads."

Scheduled Dropout Rate

The authors found that applying a constant dropout rate is not optimal. Instead, they introduced a unique V-shaped dropout scheduler. This approach challenges traditional methods like Curriculum Dropout (where dropout increases over time) by emphasizing heavy regularization at the very beginning of training.

Comparison of the proposed V-shaped dropout schedule against curriculum and anti-curriculum schedules.

The V-shaped schedule operates in three distinct phases:

  1. Early Stage: High dropout prevents dominant heads from taking over the learning process before other heads have a chance to develop.
  2. Middle Stage: Low dropout allows the model to "fit" the training data and refine its internal representations.
  3. Late Stage: High dropout acts as a final regularizer to prevent late-stage overfitting.

Experimental Results

The researchers evaluated Scheduled DropHead on two primary tasks: Machine Translation (WMT 2014 English-German) and a suite of seven text classification benchmarks (including IMDB, Yelp, and SNLI).

Key Findings

  • Translation Gains: On the WMT14 En-De task, Scheduled DropHead improved the "Transformer-big" baseline by 0.9 BLEU points, a significant margin in translation research.
  • Classification Improvements: The method consistently outperformed standard dropout across all seven classification datasets. For instance, it improved accuracy on the IMDB dataset by approximately 1.1% over a standard Transformer.
  • Robustness to Pruning: Models trained with DropHead are much more resilient to pruning. Large percentages of heads can be removed at test time with significantly less impact on accuracy compared to standard models, suggesting the model has learned redundant, high-quality pathways.
  • Reduced Head Dominance: Qualitative analysis showed that after removing the "most important" head, models trained with DropHead saw a much smaller performance drop than baseline models.

Limitations and Future Work

While effective, the authors note that finding the exact "dip" and "peak" for the V-shaped schedule requires empirical tuning for different architectures. Future research may explore applying DropHead during the massive pre-training phase of models like BERT or testing its efficacy on complex generative tasks like summarization and dialogue systems.

Summary updated

Abstract

In this paper, we introduce DropHead, a structured dropout method specifically designed for regularizing the multi-head attention mechanism, which is a key component of transformer, a state-of-the-art model for various NLP tasks. In contrast to the conventional dropout mechanisms which randomly drop units or connections, the proposed DropHead is a structured dropout method. It drops entire attention-heads during training and It prevents the multi-head attention model from being dominated by a small portion of attention heads while also reduces the risk of overfitting the training data, thus making use of the multi-head attention mechanism more efficiently. Motivated by recent studies about the learning dynamic of the multi-head attention mechanism, we propose a specific dropout rate schedule to adaptively adjust the dropout rate of DropHead and achieve better regularization effect. Experimental results on both machine translation and text classification benchmark datasets demonstrate the effectiveness of the proposed approach.
BibTeX
@inproceedings{zhou2020scheduled,
  author = {Wangchunshu Zhou and Tao Ge and Furu Wei and Ming Zhou and Ke Xu},
  title = {{Scheduled DropHead: A Regularization Method for Transformer Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=XIb6orgNyB},
}
Figure