Introduction
The original Transformer architecture, introduced in the seminal paper "Attention is All You Need," utilizes a sequential arrangement of layers. In this standard design, known as Series Attention and Feed-Forward (SAF), self-attention is computed first, its output is added to the input via a residual connection, and the resulting sum is then passed into a Feed-Forward Network (FFN).
However, recent high-performance models like Google's PaLM and various Mesh-Transformers have adopted a Parallel Attention and Feed-Forward (PAF) design. In PAF, the attention block and the FFN block process the same input simultaneously, and their outputs are combined at the end of the layer. While PAF offers significant hardware efficiency and training speed advantages, it raises a fundamental question: why does it perform as well as SAF, given that the FFN in a parallel design does not "see" the updated information from the attention layer?
Methods
To investigate the mechanics of FFNs in parallel designs, the researchers pre-trained PAF versions of two industry-standard models: RoBERTa-large and BERT-large-uncased. These models were trained on a combination of English Wikipedia and the BooksCorpus.
The study aimed to test two primary hypotheses:
- The Isotropy Hypothesis: FFNs primarily function to maintain "isotropy"—preventing token embeddings from collapsing into a single point (rank collapse) as they move deeper into the network.
- The Small Residual Hypothesis: The output of the attention block is significantly smaller in magnitude than the input embedding, meaning the FFN loses very little critical information by processing the original input in parallel.
The performance of these PAF models was evaluated against standard SAF versions using the GLUE (General Language Understanding Evaluation) benchmark. The evaluation focused on a wide range of tasks, including paraphrase detection (MRPC, QQP), sentence similarity (STS-B), sentiment analysis (SST-2), and natural language inference (QNLI, MNLI).
Key Insights
FFNs as "Anti-Gravity" for Embeddings
A critical finding of the research is that without FFNs, token embeddings tend to "collapse" toward each other. By the final layers, they lose their individual identity, making complex language tasks impossible. The FFN acts as a "spacer," providing the necessary transformations to keep embeddings spread out uniformly (isotropic). This preservation of information is vital for the model's ability to distinguish between different tokens in deep layers.
The Attention "Nudge"
The study observed that the mathematical "size" (norm) of the attention block's output is consistently much smaller than the input embedding. Because attention provides only a relatively small adjustment—a "nudge"—to the data, the FFN can maintain isotropy using the original input just as effectively as it could using the attention-adjusted input. This explains why the parallelization in PAF does not result in a significant loss of model quality.
Results
The experimental results demonstrate that PAF models perform nearly identically to their series-based counterparts across the GLUE benchmark:
- BERT-Large: The PAF version achieved an average GLUE score of 89.5, compared to 89.6 for the standard SAF version—a negligible difference of 0.1%.
- RoBERTa-Large: The PAF version scored 92.2 versus the SAF's 92.8. Although the gap is slightly larger (0.6%), the authors attribute this to the PAF version being trained on significantly less data (20GB vs. 160GB), suggesting the architecture itself is not the bottleneck.
Visual analysis confirmed that while models without FFNs suffer from near-zero isotropy (complete collapse) within a few layers, PAF models successfully maintain high isotropy throughout the network, mirroring the behavior of traditional SAF models.
Conclusion
This research clarifies why parallel Transformer designs are effective. By identifying the FFN's role in maintaining embedding isotropy and noting the relatively small scale of attention residuals, the authors provide a theoretical foundation for using PAF architectures. This allows developers to leverage the training speedups of parallel designs on modern hardware without sacrificing model intelligence. Future work may extend these findings to decoder-only models and generative tasks to see if these hypotheses hold over long-form text generation.
Summary updated