Abstract
Large Language Models (LLMs) typically rely on autoregressive decoding, a process that is computationally expensive and memory-bandwidth intensive. We present Adaptix, a fine-tuning-free framework designed to accelerate LLM decoding by up to 2.5x. Adaptix utilizes a dynamic tri-gram matrix updated in real-time alongside a Monte Carlo Tree Search (MCTS) draft maker to generate high-quality candidate tokens. These candidates are verified by the target LLM in a single forward pass using tree attention. Our results across multiple benchmarks demonstrate significant speedups and improved token acceptance rates while maintaining extreme memory efficiency compared to existing retrieval-based methods.
Introduction
The standard autoregressive decoding paradigm in LLMs—predicting one token at a time—creates a significant bottleneck in real-time applications. While techniques like Speculative Decoding attempt to mitigate this by using smaller "draft" models to predict multiple tokens ahead, they often require expensive training of auxiliary models or rely on static retrieval databases that fail to adapt to specific contexts.
Adaptix addresses these limitations by introducing an adaptive, search-based drafting system. Unlike prior methods, it does not require fine-tuning and dynamically adjusts its drafting strategy based on the model's actual output history.
The Adaptix Framework
Adaptix consists of three core components that work in tandem to optimize the drafting process:
- Adaptive Tri-gram Matrix: Instead of a complex neural network, Adaptix employs a lightweight tri-gram matrix to track token sequences. This matrix is initialized from a corpus but is updated dynamically during inference. As the LLM generates accepted tokens, the tri-gram statistics are refined, allowing the drafter to align more closely with the model's specific distribution for the current task.
- MCTS Draft Maker: To navigate the potential sequences suggested by the tri-gram matrix, Adaptix uses Monte Carlo Tree Search (MCTS). This approach balances the exploitation of high-probability sequences with the exploration of diverse paths, ensuring the generated drafts are both likely and varied.
- Tree Attention Verification: Multiple draft sequences are organized into a tree structure. The target LLM then verifies all branches of this tree simultaneously in one forward pass. This "tree attention" mechanism maximizes the number of tokens that can be accepted in a single cycle, significantly reducing total latency.
Experimental Evaluation
We evaluated Adaptix using a variety of models, including Vicuna (7B, 13B, 33B) and LLaMA2-chat (7B, 13B). Performance was measured across three major benchmarks: MT-Bench (general reasoning), Alpaca (natural language tasks), and HumanEval (Python code generation). Initial tri-gram distributions were distilled from conversational (UltraChat) and code-based (The Stack) datasets.
Results and Discussion
Adaptix demonstrates robust performance gains across all tested models and benchmarks.
Performance Gains
As shown in the results, Adaptix achieves up to a 2.5x speedup in decoding latency. In structured tasks like code generation (HumanEval), speedups reached as high as 2.69x. The average token acceptance rate improved by approximately 20% over baselines such as Lookahead Decoding and REST.
Efficiency and Robustness
A key advantage of Adaptix is its memory efficiency. It requires only ~574MB of memory, a fraction of the 12GB needed by state-of-the-art retrieval methods like REST. Furthermore, the system is robust to variations in sampling parameters like "top-p" and "temperature," maintaining its speedup advantages where other speculative methods often falter.
Conclusion
Adaptix provides a highly efficient, adaptive solution for accelerating LLM inference without the need for additional model training. By combining real-time tri-gram updates with MCTS, it offers a scalable path toward more responsive AI systems. Future work will focus on scaling these techniques for massive production environments and exploring higher-order n-gram structures for even greater drafting accuracy.
Summary updated