CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers

Longwei Zou, Qingyang Wang, Han Zhao, Jiangangkong Jiangangkong, Yi Yang, Yangdong Deng

Abstract

The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language models are reliant on an exponentially increasing number of parameters. The overwhelming computation complexity incurs a high inference latency that negatively affects user experience. Existing methods to improve inference efficiency, such as tensor parallelism and quantization, target to reduce per-layer computing latency, yet overlook the cumulative latency due to the number of layers. Recent works on reducing the cumulative latency through layer removing, however, lead to significant performance drop. Motivated by the similarity of inputs among adjacent layers, we propose to identify quasi-independent layers, which can be concurrently computed to significantly decrease inference latency. We also introduce a bypassing technique to mitigate the effect of information loss. Empirical experiments of the proposed approach on the LLaMA models confirm that Concurrent Computation of Quasi-Independent Layers (CQIL) can reduce latency by up to 48.3% on LLaMA-33B, while maintaining a close level of performance.

Introduction

The "Large" in Large Language Models comes with a heavy price: inference latency. As models grow, they add more layers, and because these layers are typically connected in a strict sequence, the time it takes to generate a single token adds up. Existing solutions generally fall into two camps: per-layer optimization (like quantization) which speeds up individual layers but maintains sequential execution, and layer removal, which often leads to a massive "performance cliff."

Observations of deep models reveal a specific phenomenon: the input to one layer is often nearly identical to the input of the next. This similarity suggests that many layers are "quasi-independent" and do not strictly require the output of the preceding layer to begin computation. This paper introduces CQIL, a framework that identifies these layers and computes them in parallel across multiple GPUs to slash cumulative latency.

Methodology

The core of the framework is the Concurrent Computation of Quasi-Independent Layers (CQIL). By analyzing cosine similarity, the authors demonstrated that as models get deeper, adjacent layers start seeing almost the same data. This supports an "ensemble" hypothesis: while early layers act as a pipeline for feature extraction, middle and deep layers act more like an ensemble that can be refined in parallel.

Architectural comparison of standard pipeline execution, CQIL, and CQIL with Bypassing.

To prevent information loss when layers are computed simultaneously, a bypassing technique is employed. This mechanism sends the attention output from the first parallel layer to the feedforward module of the second, maintaining a shortcut for data flow. To bridge any remaining performance gap, the models undergo Low-Rank Adaptation (LoRA) fine-tuning on a small subset of the RedPajama dataset (0.5B tokens), allowing the model to adapt to the parallel execution mode.

Evaluation and Results

The researchers evaluated CQIL using the LLaMA-1 family (7B, 13B, and 33B) across standard benchmarks like MMLU, HellaSwag, and ARC. The results indicate that CQIL is highly effective, particularly for larger models which possess more quasi-independent layers.

  • Latency Reduction: CQIL achieved a 48.3% speedup on LLaMA-33B using 4 GPUs and a 36% speedup on LLaMA-7B.
  • Performance Retention: On LLaMA-33B, the average score across downstream tasks only dipped slightly from 71.7% to 69.3%.
  • Small Batch Efficiency: Unlike standard Tensor Parallelism (TP), which often struggles to provide speedups for small batch sizes, CQIL provides consistent latency reduction even at a batch size of one.
Latency reduction ratio of CQIL compared to Tensor Parallelism (TP) across different batch sizes.

Conclusion and Future Work

CQIL represents an orthogonal approach to existing efficiency methods; it can be combined with quantization or tensor parallelism for additive benefits. While it requires multiple GPUs and does not reduce total FLOPs, it significantly improves the user experience by reducing wall-clock time. Future work will focus on deeper integration with other parallelization strategies and addressing the sensitivity of the model's initial and final layers, which currently must remain sequential.

Summary updated

Abstract

The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language models are reliant on an exponentially increasing number of parameters. The overwhelming computation complexity incurs a high inference latency that negatively affects user experience. Existing methods to improve inference efficiency, such as tensor parallelism and quantization, target to reduce per-layer computing latency, yet overlook the cumulative latency due to the number of layers. Recent works on reducing the cumulative latency through layer removing, however, lead to significant performance drop. Motivated by the similarity of inputs among adjacent layers, we propose to identify quasi-independent layers, which can be concurrently computed to significantly decrease inference latency. We also introduce a bypassing technique to mitigate the effect of information loss. Empirical experiments of the proposed approach on the LLaMA models confirm that Concurrent Computation of Quasi-Independent Layers (CQIL) can reduce latency by up to 48.3% on LLaMA-33B, while maintaining a close level of performance.
BibTeX
@inproceedings{zou2024cqil,
  author = {Longwei Zou and Qingyang Wang and Han Zhao and Jiangangkong Jiangangkong and Yi Yang and Yangdong Deng},
  title = {{CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=n6SEzUQgZh},
}
Figure