Abstract
Layer Parallelism (LP) is a novel inference optimization that restructures the computational graph of pre-trained Large Language Models (LLMs) to execute consecutive layers in parallel. By reducing inter-GPU communication overhead, LP achieves significant throughput gains—up to 1.19x on Llama 2 7B—with minimal impact on benchmark accuracy. This post-hoc method requires no retraining from scratch and directly addresses the "communication wall" in high-performance distributed inference.
The Communication Bottleneck in LLM Inference
Large Language Models (LLMs) are notoriously resource-intensive. While research has shown that these models are often "over-parameterized"—allowing for layer skipping or pruning with minimal logic impact—translating this redundancy into actual wall-clock speedups remains a challenge.
The primary bottleneck in modern high-performance inference is often not raw computation (FLOPs), but communication. In multi-GPU setups using Tensor Parallelism, GPUs must synchronize data after every single layer. This constant stopping and starting creates a "communication wall." Layer Parallelism (LP) tackles this by asking: if layers are somewhat independent, can we evaluate them simultaneously to skip half the synchronization points?
Methods: Restructuring the Computational Graph
The core of LP is the Contiguous 2-Parallel technique. Instead of the standard sequential processing where Layer $k+1$ waits for the output of Layer $k$, LP modifies the graph to evaluate pairs of layers simultaneously.
In a multi-GPU environment, this restructuring allows the system to perform a single "all-reduce" operation (the synchronization step) for two layers instead of one for each. A significant advantage of this approach is that it is applied post-hoc to already trained models. While it doesn't require training from scratch, a "light" fine-tuning phase (approximately 4,000 steps) can be used to recover accuracy lost during the parallelization process.
Key Insights and Robustness
Experiments across model families like Llama 2, Llama 3.2, and Qwen3 revealed several critical insights:
- Intermediate Redundancy: While the initial and final layers of an LLM are critical for performance, the intermediate layers are remarkably robust. Shuffling these middle layers often results in negligible changes to model perplexity.
- Task Sensitivity: General language tasks (measured via MMLU) remained stable under LP. However, mathematical reasoning (GSM-8K) showed higher sensitivity, suggesting that "math logic" relies on specific sequential circuits that are more fragile when parallelized.
- Scaling Benefits: Larger models, such as Qwen3 14B, demonstrated greater robustness to LP than smaller models, suggesting that "effective depth" becomes more flexible as parameter counts increase.
Results: Speed vs. Accuracy
On Llama 2 7B, the method yielded a 1.19x speedup in total throughput. In more aggressive 4-GPU configurations, the system reached up to 1.46x relative speed.
As shown in the Pareto fronts above, the trade-off for this speed is a minor dip in performance. For Llama 2 7B, reducing the effective depth from 32 to 26 layers resulted in an average accuracy drop of only 1.5% across standard benchmarks. Furthermore, synchronization efficiency in 2-GPU setups improved by nearly 2x, reducing sync time from 100.8ms to 50.7ms.
Limitations and Future Work
While promising, LP has limitations. Smaller models (under 3B parameters) see reduced benefits due to tighter layer coupling. Additionally, there is currently no theoretical framework to predict the optimal layer groupings, requiring empirical ablation. Future research may explore combining LP with quantization (INT4/INT8) to simultaneously reduce both the frequency and the size of GPU synchronizations, potentially leading to even more dramatic production speedups.
Summary updated