The standard Transformer architecture is inherently sequential. Inside a traditional Transformer layer, data flows through a strict pipeline: first, the input passes through a multi-head self-attention (MHA) module, and only after that output is computed and normalized does it enter the feed-forward network (FFN) or multi-layer perceptron (MLP).
This sequential dependency creates a major engineering bottleneck. Hardware must wait for attention operations to finish before starting the FFN, leading to frequent GPU kernel launches and idle processor cycles. In distributed settings, this structure requires separate synchronization steps (such as All-Reduce operations) for each sub-block, severely limiting training and inference throughput.
To overcome this, a prominent research direction focuses on parallel layer execution—rearchitecting the Transformer block so that attention, feed-forward, and routing modules execute concurrently rather than sequentially.
Fusing Operations via Intra-Block Parallelism
The core intuition behind intra-block parallelism is simple: instead of feeding the output of the attention layer into the MLP, both blocks can process the same incoming representation simultaneously. Their independent outputs are then summed at the end of the block.
$$\text{Standard Layer: } x_{t+1} = x_t + \text{MLP}(\text{LN}(x_t + \text{Attn}(\text{LN}(x_t))))$$ $$\text{Parallel Layer: } x_{t+1} = x_t + \text{Attn}(\text{LN}(x_t)) + \text{MLP}(\text{LN}(x_t))$$
Why does this work so well in practice? Because both modules share the exact same input vector, their initial linear transformations (the projection matrices for Queries, Keys, Values in attention, and the gate/up-projections in the MLP) can be fused into a single, massive matrix multiplication.
Standard Sequential Block:
Input ──> LayerNorm ──> Attention ──> LayerNorm ──> MLP ──> Output
Parallel Block (Fused):
┌──> Attention ──┐
Input ──┴──> MLP ────────┼──> Sum ──> Output
└──> (Other) ────┘
This structural shift yields massive efficiency gains across different scales and modalities:
- Distributed Training Efficiency: In GPT-NeoX-20B, Gao demonstrated that computing attention and FFN layers in parallel reduces the frequency of all-reduce communication operations across GPUs, increasing training throughput by approximately 15% with negligible impact on model quality.
- Vision Transformers at Scale: When scaling vision models to extreme sizes, Dehghani et al. utilized parallel attention and MLP blocks to build ViT-22B. Fusing these linear operations maximized hardware utilization and drastically improved training speed on specialized accelerators.
- Inference Latency Reduction: For real-time applications where small batch sizes are common, Touvron et al. showed that parallelizing these sub-blocks significantly cuts inference latency in Vision Transformers without sacrificing accuracy.
- Architectural Simplification: Pushing this concept to its logical conclusion, He et al. showed in Simplifying Transformer Blocks that removing redundant normalization layers and skip connections entirely while summing parallel branches allows models to train 16% faster with 15% fewer parameters.
Hiding Communication Latency in Distributed Systems
Beyond operator fusion on a single chip, decoupling the sequential dependency between attention and feed-forward modules allows distributed systems to overlap heavy network communication with active computation.
In large-scale architectures like Mixture-of-Experts (MoE) or long-context models, tokens must be dynamically routed across different nodes via expensive network operations (such as All-to-All communication). If the architecture is strictly sequential, the entire system stalls while waiting for these network transfers to complete.
Parallel execution models break this bottleneck by running a dual-stream pipeline:
- Decoupling Attention and Routing: In First Attentions Last (FAL), Kim et al. eliminated distributed training bottlenecks by reusing early attention outputs across subsequent blocks, allowing the MLP and attention modules to execute concurrently on hardware without waiting for sequential synchronization.
- Hiding MoE Routing Overhead: In MoE models, Zhang et al. demonstrated in BAM! that concurrent execution effectively hides the additional latency of complex, soft-routed expert mechanisms behind the compute-heavy feed-forward operations. Taking this further, Cai et al. developed ScMoE, which uses representations from preceding layers to trigger token routing early. This achieves up to 100% communication-computation overlap by processing self-attention and shared experts in parallel with specialized expert routing.
- Optimizing Hybrid Architectures: This paradigm is equally vital when combining different layer types. In a systematic analysis of hybrid language models, Bae et al. discovered that "intra-layer" hybrids—which run attention heads and State Space Model (SSM) heads in parallel within a single block—offer a vastly superior performance-to-efficiency tradeoff compared to alternating them in sequential blocks.
Why It Works: The Massively Parallel Computation Perspective
A natural question arises: why can we remove these internal sequential dependencies without breaking the model's ability to reason?
The answer lies in how Transformers process information globally. Theoretical work by Sanford et al. in Transformers, parallel computation, and logarithmic depth formally maps the Transformer architecture to the Massively Parallel Computation (MPC) model.
In this framework, each self-attention layer functions as a single round of global communication (routing data across tokens), while the MLPs act as local processors executing computations on individual tokens. Because the fundamental bottleneck in solving complex algorithmic or graph tasks is the number of global communication rounds (represented by the total depth of the network) rather than strict step-by-step sequential dependency within a single block, executing attention and MLP operations concurrently preserves the network's expressive power.
Further studies by Liu et al. and Back de Luca et al. show that even when attention patterns are data-agnostic or highly approximated, a fixed number of parallel rounds can perfectly simulate complex distributed algorithms like sorting or graph connectivity.
This theoretical flexibility is visible in empirical observations of trained models. As Nain et al. observed in Transformer Layers as Painters, the middle layers of large language models operate less like a rigid, step-by-step assembly line and more like a collaborative pool of parallel processors. Because these layers share a common representational space, they can be reordered, skipped, or executed in parallel with minimal degradation in accuracy—proving that intra-layer parallelism aligns naturally with how Transformers inherently process data.
Key Foundational Work
Transformer in action: a comparative study of transformer-based acoustic models for large scale speech recognition applications (2020) — contributed 1 concept elements to this direction.
Interaction retardée dans l'encodeur du Transformer pour répondre efficacement aux questions dans un domaine ouvert (2021) — contributed 1 concept elements to this direction.
Modeling Task-Aware MIMO Cardinality for Efficient Multilingual Neural Machine Translation (2021) — contributed 1 concept elements to this direction.