Decoding with Limited Teacher Supervision Requires Understanding When to Trust the Teacher

Hyunjong Ok, Jegwang Ryu, Jaeho LeeEMNLP 2024 · DOI

Introduction

Large Language Models (LLMs) have set new benchmarks in complex reasoning but remain computationally expensive and slow for edge deployment. Conversely, small language models (sLLMs) offer efficiency but often struggle with multi-step logic. While collaborative frameworks like speculative decoding exist, they typically require both models to reside in memory simultaneously—a significant constraint for many hardware environments.

This research explores a "limited supervision" paradigm: can a powerful teacher model significantly improve an sLLM's performance by assisting only with the first few tokens of a response? The authors demonstrate that even minimal intervention can yield substantial gains, provided the system knows when and how much to trust the teacher's output.

Methodology: Adaptive Token Aggregation

The core of the proposed approach is a framework that aggregates the logits (raw prediction scores) of a student model ($f_s$) and a teacher model ($f_t$). This is controlled by a tunable parameter, $\alpha$, which determines the degree of teacher influence:

$$S_\alpha = \sigma(f_s(x)) + \alpha(\sigma(f_t(x)) - \sigma(f_s(x)))$$

In this formulation, $\alpha = 0$ relies solely on the student, while $\alpha = 1$ follows the teacher. Values of $\alpha > 1$ represent "overtrusting" the teacher, a technique often used in Contrastive Decoding to penalize student-specific biases.

Overview of the limited teacher supervision decoding process.

The researchers found that a fixed $\alpha$ is suboptimal because the ideal balance varies by task and prompt. To address this, they developed two adaptive strategies:

  1. Entropy-based Knowledge Injection: The teacher is invoked only when the student's prediction entropy falls within a specific "sweet spot"—neither too confident (where the teacher adds noise) nor too uncertain (where the teacher might confuse the student).
  2. Optimal $\alpha$ Predictor: Small auxiliary models (XGBoost or a 5-layer DNN) are trained to predict the best $\alpha$ for a given token based on the logits and entropy of both models.

Experimental Results

The method was evaluated across a wide range of benchmarks, including math (GSM8K, MultiArith), science (ARC), and common-sense reasoning (StrategyQA).

The Impact of Minimal Supervision

A striking finding is the "Power of One Token." Providing just the first token from a teacher model ($N=1$) can lead to significant accuracy jumps. For instance, Phi-3-mini's performance on StrategyQA improved by 2% simply by receiving the correct initial word.

Variance in Optimal Trust

Empirical results show that the relationship between $\alpha$ and performance is highly non-linear and task-dependent. As shown in the figure below, the optimal $\alpha$ can vary significantly across different benchmarks and model pairings.

Performance across various benchmarks as a function of the trust parameter alpha.

Notably, the researchers observed cases of "Negative Alpha," where the student is better aligned with the context than the teacher. In these instances, the best strategy is to move away from the teacher's prediction.

Performance and Efficiency

The adaptive DNN-based predictor allowed Phi-3-mini to reach 82.44% accuracy on GSM8K, outperforming the 80.36% baseline and fixed-parameter strategies. Furthermore, the method is highly efficient; on an A100 GPU, it achieved a generation speed of 2.63 seconds per sentence, nearly 6x faster than Chain-of-Thought (CoT) decoding.

Conclusion and Future Work

This work demonstrates that limited teacher supervision is a viable path for boosting sLLM performance without the overhead of full speculative decoding. By adaptively managing trust through entropy and auxiliary predictors, sLLMs can leverage the strengths of larger models more effectively.

Future research will focus on overcoming tokenizer mismatches between different model families and exploring deeper architectural features beyond entropy to further refine the trust mechanism. Additionally, reducing the impact of network latency for remote teacher calls remains a key practical challenge for real-world deployment.

Summary updated

BibTeX
@inproceedings{ok2024decoding,
  author = {Hyunjong Ok and Jegwang Ryu and Jaeho Lee},
  title = {{Decoding with Limited Teacher Supervision Requires Understanding When to Trust the Teacher}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.693},
}
Figure