Enhanced Transformer Architecture for Natural Language Processing

Woohyeon Moon, Taeyoung Kim, Bumgeun Park, Dongsoo Har

Abstract

The Transformer is a state-of-the-art model in the field of natural language processing (NLP). Current NLP models primarily increase the number of transformers to improve processing performance. However, this technique requires significant training resources, such as computing capacity. In this paper, a novel structure of the Transformer is proposed, featuring full layer normalization, weighted residual connections, positional encoding exploiting reinforcement learning, and zero masked self-attention. The proposed model, termed the Enhanced Transformer, is validated using the bilingual evaluation understudy (BLEU) score on the Multi30k translation dataset. The Enhanced Transformer achieves a 202.96% higher BLEU score compared to the original Transformer.

1. Introduction

The current landscape of Natural Language Processing (NLP) is defined by a scaling "arms race." To achieve incremental performance gains, researchers typically stack more layers and increase parameter counts. However, this "bigger is better" approach comes with a massive cost in terms of computing power, energy, and training time.

This paper argues that the original Transformer architecture (Vaswani et al., 2017) contains untapped potential that can be unlocked through architectural efficiency rather than brute-force scaling. By identifying and addressing four specific bottlenecks—data distribution mismatches, vanishing gradients, redundant positional information, and self-absorption in attention—the Enhanced Transformer demonstrates that massive performance gains are possible within a similar parameter footprint.

2. Architectural Innovations

The Enhanced Transformer introduces four key modifications to the standard architecture to improve stability and contextual representation.

2.1 Full Layer Normalization

In standard architectures, input embeddings and positional encodings often have different distributions, which can make early training difficult. The Enhanced Transformer adds an extra normalization layer at the very beginning of the model. This ensures that both components are on the same scale before they are combined, leading to more stable training and lower sensitivity to initialization noise.

2.2 Weighted Residual Connection

Standard residual connections use the formula $H(x) = F(x) + x$. The authors introduce a hyperparameter $k$, modifying the formula to $H(x) = F(x) + kx$. Empirical results indicate that $k=4$ acts as a "broader roadway" for gradient flow, allowing for much stronger signal propagation in deep networks compared to the traditional identity mapping.

2.3 RL-based Positional Encoding

Rather than using fixed sinusoidal waves, the Enhanced Transformer utilizes a Soft Actor-Critic (SAC) reinforcement learning agent to "learn" optimal positional vectors. This method creates encodings that are mathematically 6.57 times more effective at distinguishing positions than traditional methods, avoiding the "dead zones" where different word positions produce nearly identical vectors.

2.4 Zero Masked Self-Attention

Standard self-attention often assigns the highest scores to the word itself (the diagonal of the attention matrix), potentially ignoring crucial contextual relationships. To force the model to focus on context, the authors implement Zero Masked Self-Attention, which manually sets the diagonal to zero. This pushes the model to find meaning in the relationships between different tokens rather than just the identity of the current token.

3. Experimental Setup and Results

The model was primarily evaluated using the Multi30k translation dataset across several language pairs, including German-to-English, Korean-to-English, Chinese-to-English, and Japanese-to-English.

3.1 Quantitative Performance

The quantitative improvements were significant. On the primary German-to-English benchmark, the original Transformer scored 18.59 BLEU, while the Enhanced Transformer reached 56.32—a 202.96% improvement. This performance also scaled to other languages, with gains of approximately 34% for Korean and Chinese, and 21% for Japanese tasks.

3.2 Comparison with State-of-the-Art

The Enhanced Transformer significantly outperformed other modern Transformer variants. With a BLEU score of 56.32, it surpassed the Post-layer normalization Transformer (46.61) and the Realformer (38.32), demonstrating that these specific architectural refinements are more effective than existing structural modifications.

4. Conclusion and Future Work

The Enhanced Transformer proves that architectural refinement is a viable and highly effective alternative to simply increasing model size. By addressing fundamental design bottlenecks, the model achieves a staggering 202% improvement in translation quality. While these results are promising on the Multi30k dataset, future work will involve testing this architecture at a multi-billion parameter scale and exploring its efficacy on generative tasks such as code synthesis and creative writing.

Summary updated

Abstract

Transformer is a state-of-the-art model in the field of natural language processing (NLP). Current NLP models primarily increase the number of transformers to improve processing performance. However, this technique requires a lot of training resources such as computing capacity. In this paper, a novel structure of Transformer is proposed. It is featured by full layer normalization, weighted residual connection, positional encoding exploiting reinforcement learning, and zero masked self-attention. The proposed Transformer model, which is called Enhanced Transformer, is validated by the bilingual evaluation understudy (BLEU) score obtained with the Multi30k translation dataset. As a result, the Enhanced Transformer achieves 202.96% higher BLEU score as compared to the original transformer with the translation dataset.
BibTeX
@inproceedings{moon2023enhanced,
  author = {Woohyeon Moon and Taeyoung Kim and Bumgeun Park and Dongsoo Har},
  title = {{Enhanced Transformer Architecture for Natural Language Processing}},
  booktitle = {PACLIC},
  year = {2023},
  url = {https://openreview.net/forum?id=M6BF9vOSYE},
}
Figure