Highway Transformer: Self-Gating Enhanced Self-Attentive Networks

Yekun Chai, Jin Shuo, Xinwen Hou

Introduction

The Transformer architecture has fundamentally changed sequence modeling through Multi-Head Self-Attention (MHSA). However, standard Transformers primarily focus on object-wise relationships—how tokens relate to one another across a sequence. They often overlook feature-wise importance, which refers to the internal significance of specific dimensions within a single token's representation. Additionally, deep Transformers can suffer from slow optimization despite residual connections.

The "Highway Transformer" addresses these challenges by introducing Self-Dependency Units (SDUs). Inspired by the gating mechanisms of LSTMs, SDUs create a "pseudo-information highway" that regulates feature-wise information flow, leading to faster convergence and improved performance in language modeling tasks.

Architecture and Methodology

The core innovation of the Highway Transformer is the SDU, a gating mechanism designed to modulate input features based on their internal dependencies.

The Self-Dependency Unit (SDU)

For an input $X$, the SDU computes a transform gate $T(X)$ using a linear projection followed by a non-linear activation (typically Sigmoid or Tanh). The input is then modulated via an element-wise product: $$SDU(X) = T(X) \odot (XW_2 + b_2)$$ This allows the model to dynamically filter information at the feature level.

Integration into the Transformer Block

SDUs are integrated as parallel branches alongside the standard sublayers of a Transformer block, specifically the Self-Attention (SA) and Feed-Forward Network (FFN) components.

Architecture of the Highway Transformer components: (a) Multi-head dot product self-attention, (b) Self-dependency units (SDU), and (c) the SDU-augmented Transformer block.

Instead of replacing the attention mechanism, the SDU's output is summed with the attention (or FFN) output before the standard "Add & Norm" step. This hybrid approach enables the model to simultaneously capture global context through attention and internal feature importance through gating.

Experimental Evaluation

The researchers evaluated the Highway Transformer on standard language modeling benchmarks, including Penn Treebank (PTB) at both character and word levels, and the larger enwik8 dataset using the Transformer-XL architecture.

Performance and Convergence

A key finding is the significant acceleration in learning. The Highway Transformer variants consistently reach lower loss levels earlier in the training process compared to standard Transformer baselines.

Training and validation bits-per-character (bpc) curves comparing the standard Transformer-L3 baseline with SDU-enhanced variants.

On character-level PTB, a 3-layer Highway Transformer with Tanh gates achieved an 8.76% improvement in perplexity. When integrated with a 6-layer Transformer-XL on enwik8, the $\sigma$-SDU variant improved the bits-per-character (bpc) by 3.1% on the development set.

Efficiency and Overhead

Despite the added complexity of gating units, the computational overhead remains minimal. For a 6-layer model, training time increased by only 1.5% to 3%, demonstrating that the Highway Transformer is a highly efficient enhancement for existing architectures.

Key Insights

  • Layer-Wise Sensitivity: Ablation studies revealed that gating is most effective in the shallower (bottom) layers of the network. These layers often handle local syntax; adding gates here helps the model optimize more rapidly toward effective feature representations.
  • Complementary Mechanisms: Attention and SDUs serve distinct purposes: attention looks across the sequence, while SDUs look within the feature vector. Their combination provides a more holistic representation.
  • Activation Functions: While Tanh gates can offer superior performance, Sigmoid ($\sigma$) gates generally provide greater stability across different configurations.

Conclusion and Future Work

The Highway Transformer successfully bridges the gap between gated recurrent logic and parallelizable self-attention. By enhancing feature-wise modeling and accelerating convergence with minimal overhead, it offers a robust upgrade for Transformer-based models. Future research may explore more parameter-efficient SDU variants, such as those utilizing depth-wise convolutions, to further optimize the "highway" for even deeper architectures.

Summary updated

Abstract

Self-attention mechanisms have made striking state-of-the-art (SOTA) progress in various sequence learning tasks, standing on the multi-headed dot product attention by attending to all the global contexts at different locations. Through a pseudo information highway, we introduce a gated component self-dependency units (SDU) that incorporates LSTM-styled gating units to replenish internal semantic importance within the multi-dimensional latent space of individual representations. The subsidiary content-based SDU gates allow for the information flow of modulated latent embeddings through skipped connections, leading to a clear margin of convergence speed with gradient descent algorithms. We may unveil the role of gating mechanism to aid in the context-based Transformer modules, with hypothesizing that SDU gates, especially on shallow layers, could push it faster to step towards suboptimal points during the optimization process.
BibTeX
@inproceedings{chai2020highway,
  author = {Yekun Chai and Jin Shuo and Xinwen Hou},
  title = {{Highway Transformer: Self-Gating Enhanced Self-Attentive Networks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=5ZPsGcXqsC},
}

Loading related content...

Figure