Tandem Transformers for Inference Efficient LLMs

Aishwarya P S, Pranav Ajit Nair, Yashas Samaga B L, Toby James Boyd, Sanjiv Kumar, Prateek Jain, Praneeth Netrapalli

TL;DR

Tandem Transformers improve Large Language Model (LLM) inference efficiency by pairing a large "primary" model with a small "secondary" model. By allowing the small model to attend directly to the large model's rich internal representations, it becomes a significantly more accurate "drafter." This architecture leads to faster generation speeds and higher performance in speculative decoding frameworks without sacrificing downstream accuracy.

The Autoregressive Bottleneck

LLMs are notoriously slow during inference because they are autoregressive: they generate text sequentially, one token at a time. This process is "memory-bound," meaning hardware accelerators spend more time moving data around than actually performing calculations.

Existing solutions like Speculative Decoding (SPEED) attempt to mitigate this by using a tiny "drafter" model to guess several tokens ahead. A large "verifier" model then checks these guesses in a single parallel pass. However, these small drafter models often lack the Natural Language Understanding (NLU) required to make high-quality guesses, especially for complex prompts. Because they operate independently, they fail to leverage the deep contextual insights the large model has already computed during the initial prompt-processing phase.

Introducing Tandem Transformers

The authors introduce a novel architecture called Tandem Transformers, consisting of a large model ($M_L$) and a small model ($M_S$). This setup bridges the gap between the drafter and the verifier.

Tandem Transformer Architecture
  1. Cross-Model Attention: Unlike standard speculative decoding where the two models are isolated, the small model in a Tandem setup is modified to "attend" to the hidden representations of the large model.
  2. Block-Mode Processing: The large model processes the prompt normally, but during generation, it operates in "block mode." It processes a chunk (block) of $\gamma$ tokens generated by the small model all at once to update its internal states.
  3. Representation Alignment: Because $M_L$ and $M_S$ have different internal dimensions, a simple linear projection layer maps $M_L$’s high-dimensional vectors into a space $M_S$ can process.
  4. Training Strategy: The most effective training regimen involves freezing the pretrained large model ($M_L$) and training only the small model ($M_S$) alongside the projection layer. The authors also utilized logit distillation, where $M_S$ learns to mimic the output probabilities of $M_L$.
  5. Adaptive Block Length: To maximize efficiency, a tiny MLP "router" is trained to predict whether the next token drafted by $M_S$ is likely to be accepted by $M_L$. If confidence is low, the system stops drafting and triggers verification early, saving compute time.

Key Insights

  • NLU vs. NLG Capacity: A core realization of this work is that "understanding" a prompt (NLU) requires significantly more model capacity than "generating" the next token (NLG). Tandem Transformers decouple these tasks: the large model handles the heavy lifting of context understanding, while the small model handles the mechanical generation, guided by the large model's "thoughts."
  • Representations > Logits: Traditional distillation only provides the small model with the final "answer" (logits) of the large model. The Tandem architecture gives the small model access to the large model's internal reasoning (hidden states), providing a much richer signal for learning.
  • The "Free" Token: When $M_L$ verifies a block, it naturally computes the representation for the next token. The authors leverage this to give $M_S$ a "free" high-quality starting token for every new block.

Datasets and Key Results

The models were trained on the PaLM2 pretraining dataset and evaluated across a wide array of standard benchmarks, including SuperGLUE, TydiQA, MBPP, WMT22, and latency benchmarks like CNN/DailyMail and Reddit Posts.

Testing the approach using Google's PaLM2 family (Bison as $M_L$, Gecko as $M_S$) yielded impressive results:

  • Accuracy Boost: A Tandem of PaLM2-Bison and PaLM2-Gecko showed a 3.3% improvement in next-token prediction accuracy over a standalone Gecko model.
  • Performance Parity at Higher Speed: The Tandem model achieved downstream performance comparable to the mid-sized PaLM2-Otter model while being 1.16x faster.
  • Speculative Decoding Gains: When integrated into the SPEED framework, the Tandem drafter was significantly more accurate, achieving a 1.14x speedup over a standard distilled Gecko drafter.
  • Acceptance Rate: On the Reddit Posts dataset, the Tandem architecture increased the per-block token acceptance rate by 11.24% compared to vanilla secondary models.
Downstream Task Performance and Speedup

Limitations and Future Work

While highly effective, the Tandem architecture has a few limitations. The authors note that while adaptive block lengths work well for single-sequence generation, they are harder to implement efficiently in high-throughput settings (large batch sizes) where different sequences might require different block lengths. Additionally, although the setup uses pretrained models, it still requires a specific fine-tuning phase to align the models.

For future directions, the authors suggest exploring whether this architecture could serve as an alternative to LoRA for efficient fine-tuning—where a large frozen model provides the "context" and a small trainable model provides the "task-specific generation." They also propose investigating if the large model could generate a high-level "plan" for the small model to execute.

Summary updated

Abstract

The autoregressive nature of conventional large language models (LLMs) inherently limits inference speed, as tokens are generated sequentially. While speculative (Leviathan et al., 2023) and parallel (Stern et al., 2018) decoding techniques attempt to mitigate this, they face limitations: either relying on less accurate smaller models for generation or failing to fully leverage the base LLM's representations. We introduce a novel architecture, Tandem transformers, to address these issues. This architecture uniquely combines (1) a small autoregressive model and (2) a large model operating in block mode (processing multiple tokens simultaneously). The small model's predictive accuracy is substantially enhanced by granting it attention to the large model's richer representations. On the PaLM2 pretraining dataset, a tandem of PaLM2-Bison and PaLM2-Gecko demonstrates a 3.3% improvement in next-token prediction accuracy over a standalone PaLM2-Gecko, offering a 1.16x speedup compared to a PaLM2-Otter model with comparable downstream performance. We further incorporate the Tandem model within the speculative decoding (SPEED) framework where the large model validates tokens from the small model. This ensures that the tandem of PaLM2-Bison and PaLM2-Gecko achieves substantial speedup (around 1.14x faster than using vanilla PaLM2-Gecko in SPEED) while maintaining identical downstream task accuracy.
BibTeX
@inproceedings{s2024tandem,
  author = {Aishwarya P S and Pranav Ajit Nair and Yashas Samaga B L and Toby James Boyd and Sanjiv Kumar and Prateek Jain and Praneeth Netrapalli},
  title = {{Tandem Transformers for Inference Efficient LLMs}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=KVy9gREVUC},
}
Figure