Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling
Abstract
Ensembling Large Language Models (LLMs) has traditionally relied on output-level ranking, which overlooks the granular probability data available during token generation. We present Generation as a Classification (GAC), a novel framework that treats next-token generation as a multi-class classification problem over a unified vocabulary. By averaging token-level distributions from multiple models and utilizing a confidence-based cascade to optimize efficiency, GAC effectively mitigates "snowballing errors" and establishes new performance benchmarks across reasoning, math, and general knowledge tasks.
What Problem Does This Paper Address?
In computer vision, ensembling multiple models to improve classification accuracy is a standard practice. In contrast, LLM ensembling typically happens at the "output level": several models generate complete responses, and a separate "ranker" selects the best one.
The authors argue that this approach is inefficient and prone to "snowballing errors." If an LLM selects an incorrect token early in a sequence, subsequent tokens often compound this error, leading to hallucinations. By the time an output-level ranker evaluates the text, the underlying logic may already be flawed. The primary technical hurdle to ensembling earlier—at the token level—is that different models use distinct tokenizers and vocabularies, making direct probability comparison difficult.
Methods: Generation as a Classification (GAC)
The GAC framework addresses the vocabulary mismatch by treating token generation as a standard multi-class classification problem across a shared "union vocabulary."
- Vocabulary Mapping: The researchers create a union vocabulary ($V^U$) encompassing all tokens from participating models (e.g., Llama-3 and Qwen). Mapping matrices ($M^i$) project each model's specific probability vector into this shared space.
- Token-Level Averaging: At each generation step, every model generates a probability distribution for the next token. These distributions are mapped to the union space and averaged. The final token is sampled from this ensemble distribution.
- Thresholded Ensembling (The Cascade): To manage the computational cost of running multiple large models, GAC employs a "gate" model. If the gate model's confidence for a token exceeds a set threshold (e.g., 0.5), its output is used directly. The full ensemble is only triggered when the primary model is uncertain, allowing for "double-checking" on difficult tokens while maintaining speed on simple ones.
Key Insights
- Tokenization Stability: Despite different tokenizers, mainstream LLMs produce identical tokens for the 5,000 most common English words over 90% of the time. This high overlap ensures the stability of the union vocabulary approach.
- Early Intervention: Token-level ensembling allows the system to "course-correct" before errors propagate. The collective probability mass of the ensemble can nudge a single model away from a hallucinated path.
- Efficiency via Uncertainty: By ensembling only the "key" tokens (often <10% of a response), GAC achieves accuracy gains comparable to full ensembling but with significantly lower latency.
Key Results
GAC consistently breaks the performance ceilings of individual state-of-the-art models. For instance, an ensemble of Qwen2-72B and Llama-3-70B achieved a 4.06% improvement over the best single model. Furthermore, GAC outperformed ranker-based methods like LLM-Blender on reasoning and math benchmarks where step-by-step precision is critical. In terms of efficiency, the thresholding method allowed an ensemble to match the performance of a heavy-duty model like Qwen1.5-72B while running faster (77.86ms/token vs. 102.11ms/token).
Limitations and Future Work
The main challenge remains the computational overhead of hosting multiple models. Additionally, managing Key-Value (KV) caches is complex when the gate model skips the ensemble; currently, non-gate models must re-process prompts to "catch up." Future research will explore partial KV cache updates to improve efficiency further.
Summary updated