TL;DR
This paper introduces Fractal-LLM, a novel method for accelerating large language model (LLM) inference by embedding compressed "fractal" layers within the original decoder. This allows for parallel generation and verification of multiple draft tokens in a single forward pass, achieving significant speedups (up to 2.47x) without sacrificing output quality or requiring additional training.
What Problem Does This Paper Address?
Autoregressive decoding, the standard method for generating text with LLMs, is notoriously slow. Each token requires a full forward pass through the model, leading to high latency, especially for long sequences. While speculative decoding methods aim to speed this up by generating multiple "draft" tokens and verifying them with fewer passes, existing approaches often rely on external draft models, require modifications to the main model's architecture (like extra heads), or still largely adhere to a token-by-token generation constraint. This paper tackles the challenge of achieving significant inference speedups while maintaining lossless output quality and avoiding the complexities of external models or architectural overhauls.
Methods
The core innovation of Fractal-LLM is the Layer Embedded Self-Compression technique. Instead of using a separate model, Fractal-LLM inserts lightweight, compressed modules (using 8-bit quantization) as "Fractal Layers" between the original decoder layers.
Here's how it works:
- Fractal Layer Insertion: These compressed modules are strategically placed within the existing decoder architecture.
- Draft Phase: When generating text, the input sequence is extended with a "draft window" of new tokens. The Fractal Layer then processes the hidden states from the preceding original layer, generating provisional logits for these draft tokens in parallel. A compressed LM head selects the top draft tokens.
- Re-embedding and Merging: The newly generated draft tokens are re-embedded using a compressed embedding layer and passed through subsequent compressed layers. The resulting hidden states are then merged back into the original hidden state, replacing only the draft token components while preserving the prefix's context. This merged state then feeds into the next original decoder layer.
- Lossless Verification: Crucially, during the verification phase, the inserted compressed network is entirely bypassed. The original, full-precision model performs a single forward pass to verify the draft tokens. If a draft token is incorrect, it's replaced by the model's prediction, and the process continues. This ensures that the final output is identical to what the original LLM would produce.
This approach allows for generating multiple tokens simultaneously within a single forward pass of the compressed components, and then verifying them with a single pass of the original model, significantly reducing the total number of sequential operations.
Datasets
The paper evaluates Fractal-LLM on several standard NLP benchmarks:
- GSM8K: A dataset of grade-school math problems.
- XSUM: A dataset for abstractive text summarization.
- CNN/DailyMail: A dataset for long-form text summarization.
- HumanEval: A benchmark for code generation tasks.
No new benchmarks were introduced; the focus was on evaluating the existing method on established datasets.
Key Insights and Ideas
- Parallelism within the Decoder: The most significant insight is that LLM inference doesn't have to be strictly token-by-token. By embedding compressed modules, we can achieve parallel draft generation within the existing decoder structure, unlocking substantial speedups.
- Lossless Compression is Key: The method's success hinges on the "lossless" aspect. The compressed components are only used for draft generation, and the final verification by the original model guarantees that the output quality is identical to the baseline.
- Trade-offs in Hyperparameter Tuning: The efficiency of Fractal-LLM is sensitive to the number of inserted Fractal Layers (
n) and the draft window size (w). Finding the optimal balance is crucial: more layers improve draft accuracy but increase computation, while a larger window can lead to more incorrect drafts, negating speedups. - Internal Embedding Advantage: Embedding compressed layers internally offers an advantage over external draft models because it can leverage high-quality intermediate representations from earlier layers of the same model, potentially leading to more accepted draft tokens per verification pass.
Key Results
- Significant Speedups: Fractal-LLM achieves substantial inference speedups, reaching up to 2.47x on the CodeLlama-13B model for HumanEval. On Llama-3 3B, speedups of 1.20x on GSM8K and 1.48x on XSUM were observed.
- Negligible Overhead: The inserted compressed model introduces minimal additional computational overhead. For instance, on the Llama-3 3B model, FLOPs increased by only about 7% while throughput improved.
- Scalability: The method scales well with model size, with larger models showing even greater relative speedups.
- Lossless Quality: Experimental results consistently show that Fractal-LLM's outputs are identical to those of the standard autoregressive baseline.
- Reduced Compute: In some cases, like with CodeLlama-13B, the method even slightly reduces total FLOPs due to the significant reduction in forward passes.
Limitations and Future Work
- KV Caching: The current evaluation does not employ Key-Value (KV) caching, which is a standard optimization in autoregressive decoding. Future work will focus on incorporating cache management for practical deployment.
- Fractal Layer Placement: The efficiency of Fractal Layers can vary based on their number and positions within the decoder. The authors plan to investigate more systematic strategies for selecting optimal placement to maximize efficiency gains.
- Draft Window Size: The paper notes that setting the draft window size (
w) too large can introduce many inaccurate tokens, increasing computational overhead. Finding the optimalwrelative to the number of Fractal Layers (n) is an ongoing area of exploration.
Summary updated