Abstract
Large Language Models (LLMs) are increasingly constrained by the "memory wall," where the memory footprint and communication bandwidth required for inference and training exceed the capabilities of modern hardware. VcLLM (Video Coded LLM) addresses this by repurposing specialized hardware video codecs, such as H.264 and H.265, to compress LLM tensors. By treating weights, gradients, and activations as "video frames," VcLLM achieves state-of-the-art compression ratios (down to ~2.9 bits per value). This approach enables models like LLaMA-3-70B to run on consumer-grade GPUs with minimal accuracy loss, while significantly reducing communication overhead.
Introduction
As LLMs scale to hundreds of billions of parameters, they encounter severe bottlenecks in memory capacity and data movement. For instance, a 340B parameter model requires nearly 700GB of VRAM, far exceeding the capacity of individual GPUs. While existing quantization methods like 4-bit rounding offer some relief, they are often specialized for specific tensor types (e.g., weights only) and require expensive calibration. VcLLM proposes a versatile alternative: utilizing the dedicated, often idle, video engines on GPUs to perform general-purpose tensor compression.
The VcLLM Framework
VcLLM maps multidimensional tensors into 4D video-like structures (time, channel, width, height) to leverage hardware-accelerated video encoders like NVIDIA’s NVENC. The compression pipeline involves several critical stages:
- Incoherent Processing: Randomized Hadamard matrices are applied to rotate tensors, spreading out extreme "outlier" values to make them more amenable to compression.
- 8-bit Quantization: Tensors are rounded to 8-bit integers, the standard input format for video codecs.
- Video Encoding: The framework utilizes intra-frame prediction and Discrete Cosine Transform (DCT). Interestingly, the authors found that while spatial patterns within a "frame" are highly compressible, temporal "inter-frame" motion prediction is counterproductive for tensors.
- Hardware Offloading: By using dedicated video engines, VcLLM preserves CUDA cores for primary model computations.
Key Insights: Tensors as Visual Data
The effectiveness of VcLLM stems from the observation that LLM tensors share spatial characteristics with video pixels. Channel-wise distributions often create "edges" and "planar blocks" that video codecs are optimized to detect. Furthermore, the DCT transform acts as a natural outlier filter, "smearing" high-magnitude values across blocks to prevent them from distorting the quantization scale.
Unlike standard fixed-bit quantization, video codecs support fractional bitrates (e.g., 2.3 bits per value), allowing for a fine-grained Pareto frontier between accuracy and memory savings. Surprisingly, the lossy compression of activations during training was found to have a "denoising" effect, potentially accelerating convergence by filtering high-frequency noise.
Experimental Results
VcLLM was evaluated on LLaMA-2, LLaMA-3, and Pythia models across eight zero-shot reasoning benchmarks. The results demonstrate that VcLLM consistently outperforms or matches specialized baselines like GPT-Q and AWQ, particularly at lower bit-widths.
Key performance metrics include:
- Consumer Hardware Compatibility: LLaMA-3-70B (128k context) was successfully run on four 8GB GPUs, with weights compressed to 2.9 bits and activations to 3.5 bits.
- Efficiency Gains: The framework achieved a 5.5x reduction in memory footprint and a 4.5x reduction in communication volume. In pipeline-parallel training, activation communication was reduced by 78%.
- Accuracy: Across benchmarks like PIQA and HellaSwag, VcLLM maintained accuracy within 1% of the full-precision baseline even at high compression ratios.
Discussion and Future Directions
The current bottleneck for VcLLM is the throughput of existing video engines, which are optimized for 4K/60fps video rather than the extreme data rates of LLM inference. The authors advocate for the development of "Tensor Codecs"—specialized hardware units that strip away video-specific features like motion estimation to provide high-bandwidth tensor compression. Such hardware could occupy as little as 0.5% of a modern GPU's die area while providing 100Gbps throughput, paving the way for the next generation of scalable AI.
Summary updated