Abstract
The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language models are reliant on an exponentially increasing number of parameters. The overwhelming computation complexity incurs a high inference latency that negatively affects user experience. Existing methods to improve inference efficiency, such as tensor parallelism and quantization, target to reduce per-layer computing latency, yet overlook the cumulative latency due to the number of layers. Recent works on reducing the cumulative latency through layer removing, however, lead to significant performance drop. Motivated by the similarity of inputs among adjacent layers, we propose to identify quasi-independent layers, which can be concurrently computed to significantly decrease inference latency. We also introduce a bypassing technique to mitigate the effect of information loss. Empirical experiments of the proposed approach on the LLaMA models confirm that Concurrent Computation of Quasi-Independent Layers (CQIL) can reduce latency by up to 48.3% on LLaMA-33B, while maintaining a close level of performance.
Introduction
Large Language Models (LLMs) are typically structured as a sequential pipeline of hundreds of layers, where each layer must wait for the output of the previous one. This architecture creates a significant latency bottleneck. While techniques like quantization or tensor parallelism (TP) optimize individual layer computation, they do not address the cumulative latency inherent in the sequential depth of the model. This paper introduces CQIL (Concurrent Computation of Quasi-Independent Layers), a method that parallelizes the execution of layers that exhibit high input similarity.
Methodology
The CQIL framework identifies layers that are "similar enough" to be computed simultaneously, effectively breaking the sequential chain. The approach consists of three main pillars:
- Quasi-Independent Layer Identification: By analyzing cosine similarity and "substitution sensitivity," the researchers found that middle and deep layers in LLMs often receive inputs that are nearly identical to those of their predecessors. This allows for partitioning the model into groups (e.g., 2 or 4 layers) that can be executed in parallel.
- Bypassing Technique: To maintain accuracy during concurrent execution, the authors introduced a bypassing mechanism. In a standard transformer block, the Attention module's output is added to the residual stream before entering the Feed-Forward Network (FFN). CQIL bypasses the attention output from an earlier layer in a group to the FFN of a later layer, mimicking the original information flow and reducing degradation.
- LoRA Fine-tuning: To bridge any remaining performance gap, a lightweight Low-Rank Adaptation (LoRA) fine-tuning is applied using a small 0.5B token subset of the RedPajama dataset.
Key Results
The researchers evaluated CQIL on the LLaMA-1 family (7B, 13B, and 33B) and Sheared-LLaMA models. Performance was measured across standard benchmarks including MMLU, ARC, HellaSwag, and PIQA.
- Significant Latency Reduction: On LLaMA-33B, CQIL achieved a 48.3% reduction in inference latency when using 4-layer concurrent groups.
- Accuracy Preservation: The performance impact remained minimal. For LLaMA-13B, the average downstream task score only shifted from 67.8 to 65.4, representing a favorable trade-off for a ~43% speedup.
- Advantage over Tensor Parallelism: CQIL provides consistent acceleration even at a batch size of 1, where traditional Tensor Parallelism often struggles. This makes it particularly suitable for real-time, online inference scenarios.
Discussion and Insights
The success of CQIL highlights an interesting property of LLM depth: while early layers act as a sequential pipeline for feature extraction, deeper layers behave more like an ensemble refining representations. This ensemble-like nature allows for parallelization without breaking the model's logic. Furthermore, larger models tend to exhibit higher layer similarity, making them even more amenable to CQIL optimization.
Limitations and Future Work
CQIL trades hardware resources for speed; it requires additional GPUs to handle concurrent computations and does not reduce total FLOPs or energy consumption. Currently, layer partitioning is manual, but future work aims to automate this process and explore the synergistic integration of CQIL with Tensor Parallelism for even greater efficiency.
Summary updated