Introduction
Large Language Models (LLMs) are notoriously slow during inference because of their autoregressive nature: they generate tokens sequentially, one by one. Each new token requires a full pass through the model, which is computationally expensive and fails to fully utilize modern hardware accelerators like GPUs and TPUs that are optimized for parallel processing.
Existing solutions, such as Speculative Decoding (SPEED), attempt to mitigate this bottleneck by using a tiny "drafter" model to guess several tokens ahead. A larger "verifier" model then checks these guesses in a single parallel pass. However, these tiny drafter models often lack the context and intelligence of the large model, leading to low acceptance rates. When the large model rejects the small model's guesses, the anticipated speed gains vanish. The core challenge is making a small model smart enough to predict exactly what a large model would say, without making it as slow as the large model itself.
To address this, researchers have introduced Tandem Transformers, a novel architecture that decouples the "understanding" (Natural Language Understanding, or NLU) and "generation" (Natural Language Generation, or NLG) phases of LLM inference. By allowing a small autoregressive model to attend to the rich internal representations of a large model, this hybrid approach creates a significantly more accurate drafter for speculative decoding.
The Tandem Architecture
The Tandem Transformer architecture consists of a Large Model ($M_L$) and a Small Model ($M_S$), working in concert through several key mechanisms:
- Representation Sharing: Unlike standard speculative decoding setups where the small model works in isolation, $M_S$ in a Tandem architecture is modified to attend directly to the hidden representations of $M_L$. This gives the small model a "look-back" advantage, effectively inheriting the contextual wisdom of the larger model.
- Block-Mode Processing: $M_L$ does not run for every single token. Instead, it processes the prompt and then "refreshes" its representations in parallel blocks (e.g., every $\gamma$ tokens), maximizing hardware efficiency.
- The Projection Layer: Because $M_L$ and $M_S$ have different internal dimensions, a simple linear projection layer is used to align $M_L$’s high-dimensional features into a format $M_S$ can process.
- Adaptive Block Length: The researchers trained a tiny "router" (a 2-layer MLP) to dynamically decide when $M_S$ should stop guessing and let $M_L$ verify, based on the confidence of $M_S$.
Modes of Operation
The Tandem architecture can operate in two primary modes:
- Standalone: $M_S$ generates the final output directly, heavily relying on $M_L$'s features to boost its own accuracy.
- Tandem + SPEED: $M_S$ acts as a super-charged drafter. Because it "sees" what the big model is thinking, its guesses are much more likely to be accepted by $M_L$ during the verification step.
Key Insights
The fundamental premise of the Tandem Transformer is the decoupling of NLU and NLG. The paper posits that deeply "understanding" a prompt requires massive model capacity, but "generating" the next token might not—provided the generator has access to the rich context generated by the large model.
Furthermore, the authors found a strong synergy with logit distillation. Teaching the small model to mimic the big model's output probabilities further enhances the Tandem architecture's performance.
Experimental Setup
The models were trained on the PaLM2 pretraining dataset and evaluated across a wide array of industry-standard benchmarks:
- Downstream Evaluation: SuperGLUE (reasoning), TydiQA (multilingual QA), MBPP (coding), and WMT22 (translation).
- Generative Benchmarks: SQuADv2, Natural Questions, TriviaQA, WebQuestions, and Lambada.
- Latency Evaluation: CNN/DailyMail, Reddit Posts summarization, and the 1 Billion Word Benchmark (LM1B).
Results and Performance
The Tandem architecture demonstrates significant improvements in both accuracy and inference speed:
- Accuracy Boost: A Tandem configuration of PaLM2-Bison and PaLM2-Gecko showed a 3.3% improvement in next-token prediction accuracy over a standalone PaLM2-Gecko model.
- Standalone Efficiency: Operating in standalone mode, the Tandem model achieved downstream performance comparable to the mid-sized PaLM2-Otter model, but was 1.16x faster.
- Speculative Decoding Gains: When integrated into the SPEED framework, Tandem Transformers were 1.11x to 1.17x faster than standard speculative decoding methods.
- Higher Acceptance Rates: On the Reddit summarization dataset, the Tandem drafter achieved an 11.24% higher acceptance rate per block compared to a vanilla secondary model.
- Overall Speedup: On TPUv5e hardware, a distilled Tandem + SPEED setup was 2.4x faster than the baseline PaLM2-Bison model while producing identical output quality.
Limitations and Future Work
While Tandem Transformers offer a compelling leap in inference efficiency, there are areas for future exploration. Currently, the information flow is unidirectional: the small model only looks at the large model's representations of past tokens. Future iterations might explore whether $M_L$ could provide a "plan" or look-ahead representation for future tokens.
Additionally, while the authors suggest Tandem could serve as an alternative to LoRA for efficient fine-tuning (by only updating the small model), this has yet to be fully explored. Finally, extending the impressive gains of the "Adaptive Block Length" feature from single-sequence generation to high-throughput, large-batch scenarios remains an open challenge for the research community.
Summary updated