Tandem Transformers for Inference Efficient LLMs

Aishwarya P. S., Pranav Ajit Nair, Yashas Samaga, Toby Boyd, Sanjiv Kumar, Prateek Jain, Praneeth Netrapalli

Introduction

Large Language Models (LLMs) are notoriously slow during inference because of their autoregressive nature: they generate tokens sequentially, one by one. Each new token requires a full pass through the model, which is computationally expensive and fails to fully utilize modern hardware accelerators like GPUs and TPUs that are optimized for parallel processing.

Existing solutions, such as Speculative Decoding (SPEED), attempt to mitigate this bottleneck by using a tiny "drafter" model to guess several tokens ahead. A larger "verifier" model then checks these guesses in a single parallel pass. However, these tiny drafter models often lack the context and intelligence of the large model, leading to low acceptance rates. When the large model rejects the small model's guesses, the anticipated speed gains vanish. The core challenge is making a small model smart enough to predict exactly what a large model would say, without making it as slow as the large model itself.

To address this, researchers have introduced Tandem Transformers, a novel architecture that decouples the "understanding" (Natural Language Understanding, or NLU) and "generation" (Natural Language Generation, or NLG) phases of LLM inference. By allowing a small autoregressive model to attend to the rich internal representations of a large model, this hybrid approach creates a significantly more accurate drafter for speculative decoding.

The Tandem Architecture

The Tandem Transformer architecture consists of a Large Model ($M_L$) and a Small Model ($M_S$), working in concert through several key mechanisms:

  • Representation Sharing: Unlike standard speculative decoding setups where the small model works in isolation, $M_S$ in a Tandem architecture is modified to attend directly to the hidden representations of $M_L$. This gives the small model a "look-back" advantage, effectively inheriting the contextual wisdom of the larger model.
  • Block-Mode Processing: $M_L$ does not run for every single token. Instead, it processes the prompt and then "refreshes" its representations in parallel blocks (e.g., every $\gamma$ tokens), maximizing hardware efficiency.
  • The Projection Layer: Because $M_L$ and $M_S$ have different internal dimensions, a simple linear projection layer is used to align $M_L$’s high-dimensional features into a format $M_S$ can process.
  • Adaptive Block Length: The researchers trained a tiny "router" (a 2-layer MLP) to dynamically decide when $M_S$ should stop guessing and let $M_L$ verify, based on the confidence of $M_S$.

Modes of Operation

The Tandem architecture can operate in two primary modes:

  1. Standalone: $M_S$ generates the final output directly, heavily relying on $M_L$'s features to boost its own accuracy.
  2. Tandem + SPEED: $M_S$ acts as a super-charged drafter. Because it "sees" what the big model is thinking, its guesses are much more likely to be accepted by $M_L$ during the verification step.

Key Insights

The fundamental premise of the Tandem Transformer is the decoupling of NLU and NLG. The paper posits that deeply "understanding" a prompt requires massive model capacity, but "generating" the next token might not—provided the generator has access to the rich context generated by the large model.

Furthermore, the authors found a strong synergy with logit distillation. Teaching the small model to mimic the big model's output probabilities further enhances the Tandem architecture's performance.

Experimental Setup

The models were trained on the PaLM2 pretraining dataset and evaluated across a wide array of industry-standard benchmarks:

  • Downstream Evaluation: SuperGLUE (reasoning), TydiQA (multilingual QA), MBPP (coding), and WMT22 (translation).
  • Generative Benchmarks: SQuADv2, Natural Questions, TriviaQA, WebQuestions, and Lambada.
  • Latency Evaluation: CNN/DailyMail, Reddit Posts summarization, and the 1 Billion Word Benchmark (LM1B).

Results and Performance

The Tandem architecture demonstrates significant improvements in both accuracy and inference speed:

  • Accuracy Boost: A Tandem configuration of PaLM2-Bison and PaLM2-Gecko showed a 3.3% improvement in next-token prediction accuracy over a standalone PaLM2-Gecko model.
  • Standalone Efficiency: Operating in standalone mode, the Tandem model achieved downstream performance comparable to the mid-sized PaLM2-Otter model, but was 1.16x faster.
  • Speculative Decoding Gains: When integrated into the SPEED framework, Tandem Transformers were 1.11x to 1.17x faster than standard speculative decoding methods.
  • Higher Acceptance Rates: On the Reddit summarization dataset, the Tandem drafter achieved an 11.24% higher acceptance rate per block compared to a vanilla secondary model.
  • Overall Speedup: On TPUv5e hardware, a distilled Tandem + SPEED setup was 2.4x faster than the baseline PaLM2-Bison model while producing identical output quality.

Limitations and Future Work

While Tandem Transformers offer a compelling leap in inference efficiency, there are areas for future exploration. Currently, the information flow is unidirectional: the small model only looks at the large model's representations of past tokens. Future iterations might explore whether $M_L$ could provide a "plan" or look-ahead representation for future tokens.

Additionally, while the authors suggest Tandem could serve as an alternative to LoRA for efficient fine-tuning (by only updating the small model), this has yet to be fully explored. Finally, extending the impressive gains of the "Adaptive Block Length" feature from single-sequence generation to high-throughput, large-batch scenarios remains an open challenge for the research community.

Summary updated

Abstract

The autoregressive nature of conventional large language models (LLMs) inherently limits inference speed, as tokens are generated sequentially. While speculative (Leviathan et al., 2023) and parallel (Stern et al., 2018) decoding techniques attempt to mitigate this, they face limitations: either relying on less accurate smaller models for generation or failing to fully leverage the base LLM’s representations. We introduce a novel architecture, Tandem transformers, to address these issues. This architecture uniquely combines (1) a small autoregressive model and (2) a large model operating in block mode (processing multiple tokens simultaneously). The small model’s predictive accuracy is substantially enhanced by granting it attention to the large model’s richer representations. On the PaLM2 pretraining dataset, a tandem of PaLM2-Bison and PaLM2-Gecko demonstrates a 3.3% improvement in next-token prediction accuracy over a standalone PaLM2-Gecko, offering a 1.16x speedup compared to a PaLM2-Otter model with comparable downstream performance. We further incorporate the Tandem model within the speculative decoding (SPEED) framework where the large model validates tokens from the small model. This ensures that the tandem of PaLM2-Bison and PaLM2-Gecko achieves substantial speedup (around 1.14x faster than using vanilla PaLM2-Gecko in SPEED) while maintaining identical downstream task accuracy.
BibTeX
@inproceedings{s2024tandem,
  author = {Aishwarya P. S. and Pranav Ajit Nair and Yashas Samaga and Toby Boyd and Sanjiv Kumar and Prateek Jain and Praneeth Netrapalli},
  title = {{Tandem Transformers for Inference Efficient LLMs}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=KVy9gREVUC},
}
Figure