Highway Transformer: Self-Gating Enhanced Self-Attentive Networks

Yekun Chai, Jin Shuo, Xinwen Hou

Abstract: Yekun Chai, Shuo Jin, Xinwen Hou. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

2020.

Introduction

While Transformer architectures have revolutionized sequence modeling through multi-head self-attention, they primarily focus on capturing relationships between tokens (object-wise dependencies). However, they often overlook the internal importance of features within a single representation. Standard Transformers rely on residual connections for gradient flow but lack the explicit gating mechanisms found in LSTMs or Highway Networks. This absence can lead to slower optimization and difficulties in capturing fine-grained, feature-level importance, particularly in the early stages of training.

The Highway Transformer addresses these limitations by introducing Self-Dependency Units (SDU)—LSTM-style gating mechanisms—into the standard Transformer architecture. By adding these gates, particularly to the shallower layers, the architecture achieves significantly faster training convergence and improved language modeling performance without heavy computational overhead.

Methodology: The Self-Dependency Unit

The core innovation is the SDU, which acts as a "pseudo information highway" running parallel to the standard attention and feed-forward layers.

SDU Mechanism

The SDU regulates a linear projection of the input using a gating function (Sigmoid or Tanh). It calculates a "transform gate" to decide which internal features are most relevant, then multiplies this by a modulated version of the input. This provides a "feature-wise" perspective, complementing the "object-wise" perspective of self-attention.

Integration and Strategic Placement

The SDU is integrated as a parallel branch. The output of a layer is computed as:
LayerNorm(Input + Attention + SDU)

Empirical studies revealed that placing these gates on the shallow (bottom) layers of deep models is most effective. These layers act as filters for local or syntactic features, allowing deeper layers to focus on complex global dependencies.

Architecture of the Highway Transformer, including (a) Multi-head self-attention, (b) Self-dependency units, and (c) the integrated SDU-augmented Transformer blocks.

Experimental Evaluation

The model was evaluated on standard language modeling benchmarks, including character-level and word-level Penn Treebank (PTB) and the larger enwik8 dataset.

Key Results

  • Performance Boost: On character-level PTB, the Highway Transformer achieved an 8.76% improvement in perplexity over a 3-layer baseline.
  • Convergence Speed: Models with SDUs exhibited a significantly sharper decline in loss during early training stages. The gated models often reached the baseline's final performance in far fewer epochs.
  • Efficiency: The computational overhead is minimal; training time increased by less than 3% (e.g., 21.16 hours vs 21.45 hours for a 6-layer model), making it a highly efficient upgrade.
Training and validation curves for loss and bits-per-character (bpc), demonstrating the faster convergence and lower final error of SDU-enhanced models.

Discussion and Conclusion

The Highway Transformer demonstrates that feature-wise gating is a powerful complement to self-attention. By selectively filtering information in the shallower layers, the model navigates the optimization landscape more efficiently. While currently focused on language modeling, the SDU represents a "cheap" yet effective upgrade for Transformer architectures, providing a better balance between local feature importance and global context. Future work may explore automated placement of these gates and their application to other domains like computer vision.

Summary updated

BibTeX
@inproceedings{chai2020highway,
  author = {Yekun Chai and Jin Shuo and Xinwen Hou},
  title = {{Highway Transformer: Self-Gating Enhanced Self-Attentive Networks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=5ZPsGcXqsC},
}
Figure