daDPO: Distribution-Aware DPO for Distilling Conversational Abilities

Zhengze Zhang, Shiqi Wang, Yiqun Shen, Simin Guo, Dahua Lin, Xiaoliang Wang, Nguyen Cam-Tu, Fei Tan

TL;DR

Smaller large language models (LLMs) typically suffer a massive drop in conversational quality compared to their larger counterparts. This paper introduces daDPO (Distribution-Aware Direct Preference Optimization), a novel white-box knowledge distillation method that integrates a teacher model's full probability distribution directly into the DPO framework. The resulting smaller models recover significant conversational ability, with a 1.5B student model occasionally even outperforming its 7B teacher.

What Problem Does This Paper Address?

As LLMs scale down to fit on edge devices or resource-constrained environments, their conversational performance plummets steeply. Shrinking a model via pruning or quantization often breaks its ability to follow complex instructions and chat naturally.

Figure 1: Conversation ability (AlpacaEval scores) drops significantly when the model gets smaller.

To fix this, researchers use Knowledge Distillation (KD) paired with Direct Preference Optimization (DPO). Typically, this is done in a "black-box" manner: the teacher model generates a good response, the student generates a weaker one, and the student is trained on this preference pair. However, this completely ignores the rich information hidden inside the teacher's full token probability distributions (the "white-box" data). Prior attempts to patch this by simply tacking a Kullback–Leibler (KL) divergence loss onto standard DPO break DPO's underlying theoretical guarantees, leading to unstable training and suboptimal performance.

Methods

The authors propose daDPO, a unified framework that mathematically blends preference optimization with distribution-based distillation.

Instead of just penalizing the student's deviation from its initial reference policy ($\pi_{ref}$), daDPO modifies the underlying Reinforcement Learning objective to include a joint constraint: one penalty to keep the student close to its reference model (ensuring training stability), and a second penalty forcing it to match the teacher's output distribution ($\pi_{te}$).

From this objective, they derive a clean, closed-form, DPO-style loss function. Crucially, daDPO operates at the sentence level rather than the token level. This design choice provides a massive practical advantage: because it doesn't rely on token-by-token matching, the teacher and student models do not need to share the same tokenizer, making cross-family model distillation possible.

Datasets

  • Training: The models were trained using a filtered and deduplicated subset of the ShareGPT dataset, focusing on the first round of each dialogue, totaling 49,839 prompts.
  • Evaluation:
  • In-domain: 300 unique, held-out prompts from ShareGPT evaluated via GPT-4 as a judge.
  • Out-of-domain: MT-Bench (multi-turn instructions), AlpacaEval (single-turn helpfulness), Arena-Hard, and AlpacaEval-2.0.
  • Academic Benchmarks: MMLU, ARC_C, GSM8K, and Hellaswag were utilized to monitor the "alignment tax" (whether chat tuning degrades core reasoning/knowledge capabilities).

Key Insights and Ideas

  • The Student's Reference Policy is an Essential Anchor: Simply replacing the student's reference model with the teacher's distribution (an approach the authors call rDPO) causes severe training instability when the student and teacher models are vastly different (e.g., a heavily pruned model). Retaining the student's original SFT model as a reference anchor is vital for stability.
  • Naive KL Tacking Fails: Directly adding a standard token-level KL loss to the final DPO loss (dDPO+KL) performs inconsistently. Incorporating the teacher's distribution directly into the underlying RL objective provides a cleaner, theoretically sound gradient signal.
Figure 3: Baselines with distribution information on Vicuna7B-20%pruned and Vicuna7B-50%pruned. It shows that daDPO consistently outperforms directly adding distillation KL loss into SFT or DPO loss.
  • Architecture Agnostic: By leveraging sentence-level probabilities, daDPO breaks a major barrier in white-box distillation, allowing researchers to transfer knowledge between entirely different model architectures (e.g., distilling from a Qwen teacher to a Llama student).

Key Results

  • Pruning Recovery: For a 20% pruned Vicuna-7B model, daDPO dramatically closed the performance gap, achieving an in-domain win rate of -7.3% compared to the teacher, vastly outperforming standard dDPO (-31.0%).
  • Punching Above Its Weight: When distilling Qwen2.5-7B into Qwen2.5-1.5B, the daDPO-enhanced student achieved a 14.0% positive win rate against its own teacher on in-domain evaluations, showing that distribution awareness helps small models occasionally outperform their creators.
  • Low Alignment Tax: Improving conversational ability did not destroy the models' core capabilities; academic task scores on MMLU and ARC_C remained steady or slightly improved compared to standard DPO baselines.

Limitations and Future Work

  • White-Box Requirement: The method still requires full access to the teacher model's logits/probabilities to compute sentence likelihoods, meaning it cannot be used with closed-source, API-only models like GPT-4 or Claude.
  • Limited to General Chat: The evaluation focused heavily on general conversational and instruction-following abilities. Whether daDPO can successfully distill complex reasoning capabilities (like those found in specialized reasoning models) remains an open question for future research.

Summary updated

Abstract

Large language models (LLMs) have demonstrated exceptional performance across various applications, but their conversational abilities decline sharply as model size decreases, presenting a barrier to their deployment in resource-constrained environments. Knowledge distillation with Direct Preference Optimization (dDPO) has emerged as a promising approach to enhancing the conversational abilities of smaller models using a larger teacher model. However, current methods primarily focus on 'black-box' KD, which only uses the teacher's responses, overlooking the output distribution offered by the teacher. This paper addresses this gap by introducing daDPO (Distribution-Aware DPO), a unified method for preference optimization and distribution-based distillation. We provide rigorous theoretical analysis and empirical validation, showing that daDPO outperforms existing methods in restoring performance for pruned models and enhancing smaller LLM models. Notably, in in-domain evaluation, our method enables a 20% pruned Vicuna1.5-7B to achieve near-teacher performance (-7.3% preference rate compared to that of dDPO's -31%), and allows Qwen2.5-1.5B to occasionally outperform its 7B teacher model (14.0% win rate).
BibTeX
@inproceedings{zhang2025dadpo,
  author = {Zhengze Zhang and Shiqi Wang and Yiqun Shen and Simin Guo and Dahua Lin and Xiaoliang Wang and Nguyen Cam-Tu and Fei Tan},
  title = {{daDPO: Distribution-Aware DPO for Distilling Conversational Abilities}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure