Enhanced Transformer Architecture for Natural Language Processing

Woohyeon Moon, Taeyoung Kim, Bumgeun Park, Dongsoo Har

Introduction

The prevailing trend in Natural Language Processing (NLP) has been the pursuit of performance through massive scaling—increasing parameters, datasets, and compute. However, this "bigger is better" approach is often computationally inefficient. Researchers at KAIST have challenged this paradigm in their paper, "Enhanced Transformer Architecture for Natural Language Processing." By refining the fundamental components of the original Transformer, they achieved a remarkable 202% improvement in translation quality without increasing the model's depth.

The Enhanced Transformer Architecture

The proposed "Enhanced Transformer" introduces four targeted architectural upgrades to the classic Vaswani et al. design. These modifications optimize internal normalization, gradient flow, spatial awareness, and attention diversity.

The standard Transformer architecture serves as the baseline for the proposed enhancements.

1. Full Layer Normalization

Standard Transformers often face training instability due to differing data distributions between input embeddings and positional encodings. The Enhanced Transformer adds normalization layers to align these distributions before they enter the main encoder/decoder blocks, leading to more stable convergence.

2. Weighted Residual Connections

In a standard residual connection, the input is added directly to the output: $H(x) = F(x) + x$. The authors introduce a weighting hyperparameter $k$, modifying the formula to $H(x) = F(x) + kx$. Empirically, $k=4$ was found to act as a "wider highway," significantly improving gradient flow during backpropagation and contributing to a nearly 60% boost in BLEU scores.

3. RL-Optimized Positional Encoding

Traditional models use fixed sinusoidal functions for positional encoding. The Enhanced Transformer utilizes a Soft Actor-Critic (SAC) reinforcement learning agent to learn an encoding scheme. This learned approach creates "smoother" representations where words in close proximity have similar embeddings, while distant words remain distinct.

Heatmap of the RL-optimized positional encoding (512x256), demonstrating the 'smoothness' of learned spatial representations.

4. Zero-Masked Self-Attention

Standard self-attention often suffers from "self-obsession," where a word pays the most attention to itself. This can overshadow relationships between different words. By masking the diagonal of the attention matrix (forcing it to zero), the model is compelled to discover more diverse and meaningful correlations across the entire sentence.

Experimental Results

The Enhanced Transformer was evaluated on the Multi30k dataset across several language pairs, including German-to-English, Korean-to-English, Chinese-to-English, and Japanese-to-English.

The results were particularly striking in German-to-English translation:

  • Performance Gain: The Enhanced Transformer achieved a BLEU score of 56.32, a 202.96% improvement over the original Transformer's score of 18.59.
  • Ablation Study: Full Layer Normalization was identified as the most impactful individual tweak, providing a 127% improvement over the baseline.
  • Comparative Performance: The model significantly outperformed other modern variants, such as Realformer (38.32 BLEU).

Discussion and Insights

The success of the Enhanced Transformer highlights several key insights:

  • Diversity in Attention: Removing self-attention (via zero-masking) forces the softmax function to distribute scores among other relevant words, increasing the richness of the learned features.
  • Learned Spatial Awareness: The RL-based positional encoding suggests that learned mathematical patterns can be superior to hard-coded sinusoidal functions for capturing linguistic structure.
  • Gradient Flow Optimization: The weighted residual connection demonstrates that simple mathematical adjustments to how signals propagate through a network can yield massive performance dividends.

Conclusion

The KAIST team's work demonstrates that the Transformer architecture still possesses significant untapped potential. By focusing on the mathematical interactions between layers rather than simply increasing layer count, they have provided a blueprint for more efficient and powerful NLP models. Future work will involve testing these enhancements on larger-scale datasets and billion-parameter models to evaluate their scalability.

Summary updated

Abstract

Transformer is a state-of-the-art model in the field of natural language processing (NLP). Current NLP models primarily increase the number of transformers to improve processing performance. However, this technique requires a lot of training resources such as computing capacity. In this paper, a novel structure of Transformer is proposed. It is featured by full layer normalization, weighted residual connection, positional encoding exploiting reinforcement learning, and zero masked self-attention. The proposed Transformer model, which is called Enhanced Transformer, is validated by the bilingual evaluation understudy (BLEU) score obtained with the Multi30k translation dataset. As a result, the Enhanced Transformer achieves 202.96% higher BLEU score as compared to the original transformer with the translation dataset.
BibTeX
@inproceedings{moon2023enhanced,
  author = {Woohyeon Moon and Taeyoung Kim and Bumgeun Park and Dongsoo Har},
  title = {{Enhanced Transformer Architecture for Natural Language Processing}},
  booktitle = {PACLIC},
  year = {2023},
  url = {https://openreview.net/forum?id=M6BF9vOSYE},
}
Figure