Introduction
The Transformer architecture has fundamentally changed sequence modeling through Multi-Head Self-Attention (MHSA). However, standard Transformers primarily focus on object-wise relationships—how tokens relate to one another across a sequence. They often overlook feature-wise importance, which refers to the internal significance of specific dimensions within a single token's representation. Additionally, deep Transformers can suffer from slow optimization despite residual connections.
The "Highway Transformer" addresses these challenges by introducing Self-Dependency Units (SDUs). Inspired by the gating mechanisms of LSTMs, SDUs create a "pseudo-information highway" that regulates feature-wise information flow, leading to faster convergence and improved performance in language modeling tasks.
Architecture and Methodology
The core innovation of the Highway Transformer is the SDU, a gating mechanism designed to modulate input features based on their internal dependencies.
The Self-Dependency Unit (SDU)
For an input $X$, the SDU computes a transform gate $T(X)$ using a linear projection followed by a non-linear activation (typically Sigmoid or Tanh). The input is then modulated via an element-wise product: $$SDU(X) = T(X) \odot (XW_2 + b_2)$$ This allows the model to dynamically filter information at the feature level.
Integration into the Transformer Block
SDUs are integrated as parallel branches alongside the standard sublayers of a Transformer block, specifically the Self-Attention (SA) and Feed-Forward Network (FFN) components.
Instead of replacing the attention mechanism, the SDU's output is summed with the attention (or FFN) output before the standard "Add & Norm" step. This hybrid approach enables the model to simultaneously capture global context through attention and internal feature importance through gating.
Experimental Evaluation
The researchers evaluated the Highway Transformer on standard language modeling benchmarks, including Penn Treebank (PTB) at both character and word levels, and the larger enwik8 dataset using the Transformer-XL architecture.
Performance and Convergence
A key finding is the significant acceleration in learning. The Highway Transformer variants consistently reach lower loss levels earlier in the training process compared to standard Transformer baselines.
On character-level PTB, a 3-layer Highway Transformer with Tanh gates achieved an 8.76% improvement in perplexity. When integrated with a 6-layer Transformer-XL on enwik8, the $\sigma$-SDU variant improved the bits-per-character (bpc) by 3.1% on the development set.
Efficiency and Overhead
Despite the added complexity of gating units, the computational overhead remains minimal. For a 6-layer model, training time increased by only 1.5% to 3%, demonstrating that the Highway Transformer is a highly efficient enhancement for existing architectures.
Key Insights
- Layer-Wise Sensitivity: Ablation studies revealed that gating is most effective in the shallower (bottom) layers of the network. These layers often handle local syntax; adding gates here helps the model optimize more rapidly toward effective feature representations.
- Complementary Mechanisms: Attention and SDUs serve distinct purposes: attention looks across the sequence, while SDUs look within the feature vector. Their combination provides a more holistic representation.
- Activation Functions: While Tanh gates can offer superior performance, Sigmoid ($\sigma$) gates generally provide greater stability across different configurations.
Conclusion and Future Work
The Highway Transformer successfully bridges the gap between gated recurrent logic and parallelizable self-attention. By enhancing feature-wise modeling and accelerating convergence with minimal overhead, it offers a robust upgrade for Transformer-based models. Future research may explore more parameter-efficient SDU variants, such as those utilizing depth-wise convolutions, to further optimize the "highway" for even deeper architectures.
Summary updated