TL;DR
Smaller large language models (LLMs) typically suffer a massive drop in conversational quality compared to their larger counterparts. This paper introduces daDPO (Distribution-Aware Direct Preference Optimization), a novel white-box knowledge distillation method that integrates a teacher model's full probability distribution directly into the DPO framework. The resulting smaller models recover significant conversational ability, with a 1.5B student model occasionally even outperforming its 7B teacher.
What Problem Does This Paper Address?
As LLMs scale down to fit on edge devices or resource-constrained environments, their conversational performance plummets steeply. Shrinking a model via pruning or quantization often breaks its ability to follow complex instructions and chat naturally.
To fix this, researchers use Knowledge Distillation (KD) paired with Direct Preference Optimization (DPO). Typically, this is done in a "black-box" manner: the teacher model generates a good response, the student generates a weaker one, and the student is trained on this preference pair. However, this completely ignores the rich information hidden inside the teacher's full token probability distributions (the "white-box" data). Prior attempts to patch this by simply tacking a Kullback–Leibler (KL) divergence loss onto standard DPO break DPO's underlying theoretical guarantees, leading to unstable training and suboptimal performance.
Methods
The authors propose daDPO, a unified framework that mathematically blends preference optimization with distribution-based distillation.
Instead of just penalizing the student's deviation from its initial reference policy ($\pi_{ref}$), daDPO modifies the underlying Reinforcement Learning objective to include a joint constraint: one penalty to keep the student close to its reference model (ensuring training stability), and a second penalty forcing it to match the teacher's output distribution ($\pi_{te}$).
From this objective, they derive a clean, closed-form, DPO-style loss function. Crucially, daDPO operates at the sentence level rather than the token level. This design choice provides a massive practical advantage: because it doesn't rely on token-by-token matching, the teacher and student models do not need to share the same tokenizer, making cross-family model distillation possible.
Datasets
- Training: The models were trained using a filtered and deduplicated subset of the ShareGPT dataset, focusing on the first round of each dialogue, totaling 49,839 prompts.
- Evaluation:
- In-domain: 300 unique, held-out prompts from ShareGPT evaluated via GPT-4 as a judge.
- Out-of-domain: MT-Bench (multi-turn instructions), AlpacaEval (single-turn helpfulness), Arena-Hard, and AlpacaEval-2.0.
- Academic Benchmarks: MMLU, ARC_C, GSM8K, and Hellaswag were utilized to monitor the "alignment tax" (whether chat tuning degrades core reasoning/knowledge capabilities).
Key Insights and Ideas
- The Student's Reference Policy is an Essential Anchor: Simply replacing the student's reference model with the teacher's distribution (an approach the authors call
rDPO) causes severe training instability when the student and teacher models are vastly different (e.g., a heavily pruned model). Retaining the student's original SFT model as a reference anchor is vital for stability. - Naive KL Tacking Fails: Directly adding a standard token-level KL loss to the final DPO loss (
dDPO+KL) performs inconsistently. Incorporating the teacher's distribution directly into the underlying RL objective provides a cleaner, theoretically sound gradient signal.
- Architecture Agnostic: By leveraging sentence-level probabilities, daDPO breaks a major barrier in white-box distillation, allowing researchers to transfer knowledge between entirely different model architectures (e.g., distilling from a Qwen teacher to a Llama student).
Key Results
- Pruning Recovery: For a 20% pruned Vicuna-7B model, daDPO dramatically closed the performance gap, achieving an in-domain win rate of -7.3% compared to the teacher, vastly outperforming standard dDPO (-31.0%).
- Punching Above Its Weight: When distilling Qwen2.5-7B into Qwen2.5-1.5B, the daDPO-enhanced student achieved a 14.0% positive win rate against its own teacher on in-domain evaluations, showing that distribution awareness helps small models occasionally outperform their creators.
- Low Alignment Tax: Improving conversational ability did not destroy the models' core capabilities; academic task scores on MMLU and ARC_C remained steady or slightly improved compared to standard DPO baselines.
Limitations and Future Work
- White-Box Requirement: The method still requires full access to the teacher model's logits/probabilities to compute sentence likelihoods, meaning it cannot be used with closed-source, API-only models like GPT-4 or Claude.
- Limited to General Chat: The evaluation focused heavily on general conversational and instruction-following abilities. Whether daDPO can successfully distill complex reasoning capabilities (like those found in specialized reasoning models) remains an open question for future research.
Summary updated