Abstract
The autoregressive nature of conventional large language models (LLMs) inherently limits inference speed, as tokens are generated sequentially. While speculative and parallel decoding techniques attempt to mitigate this, they face limitations: either relying on less accurate smaller models for generation or failing to fully leverage the base LLM's representations. We introduce a novel architecture, Tandem transformers, to address these issues. This architecture uniquely combines (1) a small autoregressive model and (2) a large model operating in block mode (processing multiple tokens simultaneously). The small model's predictive accuracy is substantially enhanced by granting it attention to the large model's richer representations. On the PaLM2 pretraining dataset, a tandem of PaLM2-Bison and PaLM2-Gecko demonstrates a 3.3% improvement in next-token prediction accuracy over a standalone PaLM2-Gecko, offering a 1.16x speedup compared to a PaLM2-Otter model with comparable downstream performance. We further incorporate the tandem model within the speculative decoding (SPEED) framework where the large model validates tokens from the small model. This ensures that the Tandem of PaLM2-Bison and PaLM2-Gecko achieves substantial speedup (around 1.14x faster than using vanilla PaLM2-Gecko in SPEED) while maintaining identical downstream task accuracy.
Introduction
The "memory wall" and the sequential nature of Large Language Models (LLMs) make inference incredibly expensive. Standard LLMs are autoregressive: they generate one token at a time, and for each token, the entire model—often consisting of billions of parameters—must be loaded into memory. This process is highly inefficient because modern hardware, such as GPUs and TPUs, is optimized for massive parallel matrix multiplications rather than the sequential vector-matrix operations required for one-by-one token generation.
While "speculative decoding" (using a tiny "drafter" model to guess tokens that a larger "verifier" model then checks) helps mitigate this, it often relies on weak drafter models that lack the contextual understanding of the primary model. This research addresses a fundamental question: Do we require the same massive model capacity for Natural Language Understanding (NLU/prefill) as we do for Natural Language Generation (NLG/response)? Current decoder-only architectures tightly couple these two tasks, forcing the use of a high-capacity model even for the relatively simple task of generating common tokens. Tandem Transformers propose a decoupling of these functions to optimize inference efficiency.
Methodology
The Tandem Transformer architecture consists of a primary large model ($M_L$) and a secondary small model ($M_S$). The system operates through several key mechanisms:
The Tandem Loop
In this architecture, $M_L$ processes the initial prompt to create high-quality internal representations. Subsequently, $M_S$ generates a "block" of $\gamma$ tokens autoregressively. Crucially, during generation, $M_S$ does not rely solely on its own previous tokens; it attends to the rich representations generated by $M_L$ for all previous blocks. This allows the smaller model to benefit from the "knowledge" of the larger model without the computational cost of running the larger model for every token.
Block-Mode Processing and Alignment
Once $M_S$ completes a block, $M_L$ "refreshes" the context by processing those $\gamma$ tokens simultaneously in parallel. This block-mode processing is significantly faster than sequential generation. To bridge the gap between the different internal dimensions of $M_L$ and $M_S$, the architecture employs a simple linear projection layer to map $M_L$’s high-dimensional representations into a space $M_S$ can utilize.
SPEED Integration and Adaptive Routing
The authors further integrated this architecture into the SPEED (Speculative Decoding) framework, where $M_L$ acts as a verifier for the drafts produced by the "boosted" $M_S$. To further optimize the speed-accuracy tradeoff, a tiny 2-layer MLP "router" was trained to predict the likelihood of $M_S$ making an error. If the router's confidence falls below a certain threshold, it stops $M_S$ early and invokes $M_L$ to verify the sequence, ensuring high fidelity.
Experimental Results
The models were evaluated across a diverse range of benchmarks, including SuperGLUE, TydiQA, MBPP (coding), and various summarization tasks (CNN/DailyMail). The primary testing was conducted using the PaLM2 family (Bison, Otter, and Gecko).
- Accuracy Gains: A Tandem configuration of PaLM2-Bison and PaLM2-Gecko achieved a 3.3% improvement in next-token prediction accuracy compared to a standalone Gecko model.
- Inference Speedups: The Tandem model demonstrated a 1.16x speedup over a PaLM2-Otter model while maintaining comparable downstream performance. When integrated into the SPEED framework, the Tandem approach was 1.14x faster than using a standard vanilla small model as a drafter.
- End-to-End Latency: On the LM1B benchmark, the distilled Tandem + SPEED setup achieved a 2.4x speedup compared to a vanilla PaLM2-Bison model.
Discussion and Insights
The core insight of Tandem Transformers is the effective decoupling of NLU and NLG. A small model can be surprisingly accurate at generation if it is guided by the internal states of a much larger model; it does not need to store all world knowledge if it can access the large model's summary of that knowledge. By forcing the large model to operate primarily in block mode, the architecture maximizes the utilization of ML accelerators, transforming a sequential bottleneck into a parallel advantage.
While the results are promising, some limitations remain. Scaling the "Adaptive Block Length" router to work efficiently with large batches or multiple samples is a current challenge. Additionally, while freezing the large model during training is effective, the system still requires a specialized alignment phase. Interestingly, the authors suggest that Tandem Transformers could serve as an alternative to Low-Rank Adaptation (LoRA) for fine-tuning, as they allow for efficient adaptation by updating only the small model's parameters.
Conclusion
Tandem Transformers offer a path forward for efficient LLM inference by leveraging the strengths of both large and small models. By sharing representations and utilizing block-mode processing, the architecture achieves significant speedups without sacrificing the accuracy and understanding inherent in large-scale language models.
Summary updated