Using Advanced LLMs to Enhance Smaller LLMs: An Interpretable Knowledge Distillation Approach

Tong Wang, K. Sudhir, Dat HongarXiv 2024 · DOI · arXiv

Introduction

As frontier Large Language Models (LLMs) like GPT-4 continue to grow in capability, the practical challenges of deploying them—high latency, significant operational costs, and data privacy concerns—remain substantial. While smaller, "deployable" models are preferred for high-volume tasks, they often lack the nuance required for complex, multi-turn interactions. Traditional Knowledge Distillation (KD) attempts to bridge this gap by training student models to mimic the outputs of a larger teacher. However, this approach often fails in dynamic environments.

This paper introduces Guidance Elicitation and Retrieval (GER), a novel framework that shifts the focus from output imitation to strategic coaching. By distilling a library of interpretable, textual advice, GER enables smaller models to achieve high performance while remaining controllable and transparent.

The Limitations of Traditional Distillation

Standard Knowledge Distillation typically relies on Supervised Fine-Tuning (SFT), where a student model learns to predict the exact tokens generated by a teacher. The authors identify two critical flaws in this paradigm:

  1. The "Open-Loop" Problem: Conversations are inherently path-dependent. If a student model deviates slightly from the teacher's path early in a dialogue, it enters a state the teacher never encountered during training. Because the student only learned to mimic "expert" trajectories, it lacks the strategy to recover from its own errors.
  2. The "Black Box" Problem: Fine-tuning embeds behavioral changes directly into billions of numerical parameters. This makes it nearly impossible for human operators to inspect or update the model's behavior without undergoing an expensive and opaque retraining process.

The GER Framework: Distillation as Coaching

GER addresses these issues by treating distillation as the creation of an external "strategy library" rather than a simple weight update. The framework operates through three primary stages:

1. Guidance Elicitation

During the training phase, the teacher model (e.g., GPT-4) observes the student's attempts to resolve a dialogue. Instead of providing the "correct" answer, the teacher generates "textual gradients"—natural language feedback that identifies strategic deficits. For example, the teacher might advise: "Acknowledge the customer's frustration before proposing a technical solution." This feedback is iteratively refined until the student's performance stabilizes.

2. Guidance Retrieval

The refined advice is organized into a library of (state, guidance) pairs. At inference time, the system uses a Retrieval-Augmented Generation (RAG) approach. When a student encounters a new query, it retrieves the most semantically relevant coaching from the library and appends it to its prompt, allowing the model to apply the strategy in real-time.

3. Staged Curriculum

To solve the open-loop problem, GER employs a staged training curriculum. Initially, the teacher leads the conversations to establish a baseline of expert behavior. Gradually, the student takes more control, intentionally exposing the teacher to the "messy" or suboptimal states the student is likely to induce. This ensures the guidance library includes robust recovery strategies for real-world scenarios.

Experimental Evaluation

The researchers evaluated GER using goal-oriented customer service dialogues, specifically focusing on high-stakes airline ticket resolution. This domain requires strict policy enforcement alongside emotional intelligence. The study utilized a diverse range of student models, including LLaMA-2 (7B, 13B, 70B), GPT-3.5-Turbo, and reasoning-centric models like DeepSeek-R1.

Evaluation included:

  • Synthetic Benchmarks: Thousands of simulated agent-customer interactions.
  • Human Evaluation: Pairwise preference tests with 256 participants and live, interactive evaluations with 132 participants.

Key Results and Insights

  • Significant Performance Gains: GER narrowed the gap between small models and frontier models. For LLaMA-2-7B, the framework resulted in a 31 percentage point increase in human preference over the base model.
  • Superiority Over Fine-Tuning: In many instances, GER matched or outperformed Supervised Fine-Tuning, particularly in qualitative metrics like empathy and responsiveness (measured via the RATER framework).
  • Model-Agnostic Portability: Because the guidance is stored as text, it is "hot-swappable." The authors successfully applied a library built for LLaMA-2-7B to GPT-3.5 without any additional training, resulting in immediate performance improvements.
  • Interpretability and Control: Unlike traditional KD, the strategy library is human-readable. If a company's refund policy changes, managers can simply edit the text in the library to update the model's behavior, bypassing the need for retraining.

Conclusion and Future Work

GER represents a shift from surface-level mimicry to the distillation of underlying logic. While the framework requires the student model to possess a baseline level of instruction-following capability, it offers a powerful alternative for organizations requiring interpretable and robust AI. Future research will explore adaptive curriculum designs that tailor scenario generation to a student's specific learning trajectory and expand the framework into other interactive domains such as complex negotiation and sales.

Summary updated

BibTeX
@misc{wang2024using,
  author = {Tong Wang and K. Sudhir and Dat Hong},
  title = {{Using Advanced LLMs to Enhance Smaller LLMs: An Interpretable Knowledge Distillation Approach}},
  year = {2024},
  eprint = {2408.07238},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.07238},
}
Figure