VcLLM: Video Codecs are Secretly Tensor Codecs

Ceyu Xu, Yongji Wu, Xinyu Yang, Beidi Chen, Matthew Lentz, Danyang Zhuo, Lisa Wu WillsarXiv 2024 · DOI · arXiv

Abstract

Large Language Models (LLMs) are increasingly constrained by the "memory wall," where the memory footprint and communication bandwidth required for inference and training exceed the capabilities of modern hardware. VcLLM (Video Coded LLM) addresses this by repurposing specialized hardware video codecs, such as H.264 and H.265, to compress LLM tensors. By treating weights, gradients, and activations as "video frames," VcLLM achieves state-of-the-art compression ratios (down to ~2.9 bits per value). This approach enables models like LLaMA-3-70B to run on consumer-grade GPUs with minimal accuracy loss, while significantly reducing communication overhead.

Introduction

As LLMs scale to hundreds of billions of parameters, they encounter severe bottlenecks in memory capacity and data movement. For instance, a 340B parameter model requires nearly 700GB of VRAM, far exceeding the capacity of individual GPUs. While existing quantization methods like 4-bit rounding offer some relief, they are often specialized for specific tensor types (e.g., weights only) and require expensive calibration. VcLLM proposes a versatile alternative: utilizing the dedicated, often idle, video engines on GPUs to perform general-purpose tensor compression.

The VcLLM Framework

VcLLM maps multidimensional tensors into 4D video-like structures (time, channel, width, height) to leverage hardware-accelerated video encoders like NVIDIA’s NVENC. The compression pipeline involves several critical stages:

  1. Incoherent Processing: Randomized Hadamard matrices are applied to rotate tensors, spreading out extreme "outlier" values to make them more amenable to compression.
  2. 8-bit Quantization: Tensors are rounded to 8-bit integers, the standard input format for video codecs.
  3. Video Encoding: The framework utilizes intra-frame prediction and Discrete Cosine Transform (DCT). Interestingly, the authors found that while spatial patterns within a "frame" are highly compressible, temporal "inter-frame" motion prediction is counterproductive for tensors.
  4. Hardware Offloading: By using dedicated video engines, VcLLM preserves CUDA cores for primary model computations.
The VcLLM compression pipeline and bit-per-value reduction.

Key Insights: Tensors as Visual Data

The effectiveness of VcLLM stems from the observation that LLM tensors share spatial characteristics with video pixels. Channel-wise distributions often create "edges" and "planar blocks" that video codecs are optimized to detect. Furthermore, the DCT transform acts as a natural outlier filter, "smearing" high-magnitude values across blocks to prevent them from distorting the quantization scale.

Unlike standard fixed-bit quantization, video codecs support fractional bitrates (e.g., 2.3 bits per value), allowing for a fine-grained Pareto frontier between accuracy and memory savings. Surprisingly, the lossy compression of activations during training was found to have a "denoising" effect, potentially accelerating convergence by filtering high-frequency noise.

Experimental Results

VcLLM was evaluated on LLaMA-2, LLaMA-3, and Pythia models across eight zero-shot reasoning benchmarks. The results demonstrate that VcLLM consistently outperforms or matches specialized baselines like GPT-Q and AWQ, particularly at lower bit-widths.

Accuracy vs. Average Bit-Width across eight commonsense reasoning benchmarks.

Key performance metrics include:

  • Consumer Hardware Compatibility: LLaMA-3-70B (128k context) was successfully run on four 8GB GPUs, with weights compressed to 2.9 bits and activations to 3.5 bits.
  • Efficiency Gains: The framework achieved a 5.5x reduction in memory footprint and a 4.5x reduction in communication volume. In pipeline-parallel training, activation communication was reduced by 78%.
  • Accuracy: Across benchmarks like PIQA and HellaSwag, VcLLM maintained accuracy within 1% of the full-precision baseline even at high compression ratios.

Discussion and Future Directions

The current bottleneck for VcLLM is the throughput of existing video engines, which are optimized for 4K/60fps video rather than the extreme data rates of LLM inference. The authors advocate for the development of "Tensor Codecs"—specialized hardware units that strip away video-specific features like motion estimation to provide high-bandwidth tensor compression. Such hardware could occupy as little as 0.5% of a modern GPU's die area while providing 100Gbps throughput, paving the way for the next generation of scalable AI.

Summary updated

Abstract

As the parameter size of large language models (LLMs) continues to expand, the need for a large memory footprint and high communication bandwidth have become significant bottlenecks for the training and inference of LLMs. To mitigate these bottlenecks, various tensor compression techniques have been proposed to reduce the data size, thereby alleviating memory requirements and communication pressure. Our research found that video codecs, despite being originally designed for compressing videos, show excellent efficiency when compressing various types of tensors. We demonstrate that video codecs can be versatile and general-purpose tensor codecs while achieving the state-of-the-art compression efficiency in various tasks. We further make use of the hardware video encoding and decoding module available on GPUs to create a framework capable of both inference and training with video codecs repurposed as tensor codecs. This greatly reduces the requirement for memory capacity and communication bandwidth, enabling training and inference of large models on consumer-grade GPUs.
BibTeX
@misc{xu2024vcllm,
  author = {Ceyu Xu and Yongji Wu and Xinyu Yang and Beidi Chen and Matthew Lentz and Danyang Zhuo and Lisa Wu Wills},
  title = {{VcLLM: Video Codecs are Secretly Tensor Codecs}},
  year = {2024},
  eprint = {2407.00467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.00467},
}
Figure