TL;DR
Tandem Transformers improve Large Language Model (LLM) inference efficiency by pairing a large "primary" model with a small "secondary" model. By allowing the small model to attend directly to the large model's rich internal representations, it becomes a significantly more accurate "drafter." This architecture leads to faster generation speeds and higher performance in speculative decoding frameworks without sacrificing downstream accuracy.
The Autoregressive Bottleneck
LLMs are notoriously slow during inference because they are autoregressive: they generate text sequentially, one token at a time. This process is "memory-bound," meaning hardware accelerators spend more time moving data around than actually performing calculations.
Existing solutions like Speculative Decoding (SPEED) attempt to mitigate this by using a tiny "drafter" model to guess several tokens ahead. A large "verifier" model then checks these guesses in a single parallel pass. However, these small drafter models often lack the Natural Language Understanding (NLU) required to make high-quality guesses, especially for complex prompts. Because they operate independently, they fail to leverage the deep contextual insights the large model has already computed during the initial prompt-processing phase.
Introducing Tandem Transformers
The authors introduce a novel architecture called Tandem Transformers, consisting of a large model ($M_L$) and a small model ($M_S$). This setup bridges the gap between the drafter and the verifier.
- Cross-Model Attention: Unlike standard speculative decoding where the two models are isolated, the small model in a Tandem setup is modified to "attend" to the hidden representations of the large model.
- Block-Mode Processing: The large model processes the prompt normally, but during generation, it operates in "block mode." It processes a chunk (block) of $\gamma$ tokens generated by the small model all at once to update its internal states.
- Representation Alignment: Because $M_L$ and $M_S$ have different internal dimensions, a simple linear projection layer maps $M_L$’s high-dimensional vectors into a space $M_S$ can process.
- Training Strategy: The most effective training regimen involves freezing the pretrained large model ($M_L$) and training only the small model ($M_S$) alongside the projection layer. The authors also utilized logit distillation, where $M_S$ learns to mimic the output probabilities of $M_L$.
- Adaptive Block Length: To maximize efficiency, a tiny MLP "router" is trained to predict whether the next token drafted by $M_S$ is likely to be accepted by $M_L$. If confidence is low, the system stops drafting and triggers verification early, saving compute time.
Key Insights
- NLU vs. NLG Capacity: A core realization of this work is that "understanding" a prompt (NLU) requires significantly more model capacity than "generating" the next token (NLG). Tandem Transformers decouple these tasks: the large model handles the heavy lifting of context understanding, while the small model handles the mechanical generation, guided by the large model's "thoughts."
- Representations > Logits: Traditional distillation only provides the small model with the final "answer" (logits) of the large model. The Tandem architecture gives the small model access to the large model's internal reasoning (hidden states), providing a much richer signal for learning.
- The "Free" Token: When $M_L$ verifies a block, it naturally computes the representation for the next token. The authors leverage this to give $M_S$ a "free" high-quality starting token for every new block.
Datasets and Key Results
The models were trained on the PaLM2 pretraining dataset and evaluated across a wide array of standard benchmarks, including SuperGLUE, TydiQA, MBPP, WMT22, and latency benchmarks like CNN/DailyMail and Reddit Posts.
Testing the approach using Google's PaLM2 family (Bison as $M_L$, Gecko as $M_S$) yielded impressive results:
- Accuracy Boost: A Tandem of PaLM2-Bison and PaLM2-Gecko showed a 3.3% improvement in next-token prediction accuracy over a standalone Gecko model.
- Performance Parity at Higher Speed: The Tandem model achieved downstream performance comparable to the mid-sized PaLM2-Otter model while being 1.16x faster.
- Speculative Decoding Gains: When integrated into the SPEED framework, the Tandem drafter was significantly more accurate, achieving a 1.14x speedup over a standard distilled Gecko drafter.
- Acceptance Rate: On the Reddit Posts dataset, the Tandem architecture increased the per-block token acceptance rate by 11.24% compared to vanilla secondary models.
Limitations and Future Work
While highly effective, the Tandem architecture has a few limitations. The authors note that while adaptive block lengths work well for single-sequence generation, they are harder to implement efficiently in high-throughput settings (large batch sizes) where different sequences might require different block lengths. Additionally, although the setup uses pretrained models, it still requires a specific fine-tuning phase to align the models.
For future directions, the authors suggest exploring whether this architecture could serve as an alternative to LoRA for efficient fine-tuning—where a large frozen model provides the "context" and a small trainable model provides the "task-specific generation." They also propose investigating if the large model could generate a high-level "plan" for the small model to execute.
Summary updated