CCDE: A Compact and Competitive Dialogue Evaluation Framework via Knowledge Distillation of Large Language Models

Guanghui Ye, Huan Zhao, Bo Li, Haijiao Chen, Zhixue Zhao, Zhihua Jiang, Keqin LiIEEE Transactions on Computational Social Systems 2025 · OpenReview

Abstract

Automatic evaluation metrics not only play a vital role in developing dialogue and interactive systems but also have a great impact on social activities in our daily life. However, previous specialized metrics for evaluating dialogues exhibit a relatively low correlation with human judgments. In addition, today’s state-of-the-art (SOTA) evaluators that leverage large language models (LLMs) are challenging to deploy in real-world applications due to their sheer size. To this end, we propose a novel evaluation framework, compact and competitive dialogue evaluation (CCDE), which leverages knowledge distillation of LLMs to generate training data and sequentially learn a multitask evaluator regarding diversified quality dimensions. Specifically, we first employ ChatGPT as teacher to generate a high-quality and rich-annotation corpus, CCDE-data. Then, we implement a student evaluator CCDE (1.3B) via using InstructGPT as the backbone model that is trained and fine-tuned on CCDE-data. We conduct extensive experiments on three public benchmarks: fine-grained evaluation of dialog (FED), PersonaChat, and TopicalChat. The results demonstrate that our model CCDE can outperform the current SOTA model G-Eval which calls GPT-4 ($\boldsymbol{\geq}$ 175B) by 4.3 on the FED dataset, 3.5 on the PersonaChat dataset, and 0.3 on the TopicalChat dataset, in terms of the Spearman correlation metric (%). We release the data and code at: https://anonymous.4open.science/r/ccde-3827.
BibTeX
@article{ye2025ccde,
  author = {Guanghui Ye and Huan Zhao and Bo Li and Haijiao Chen and Zhixue Zhao and Zhihua Jiang and Keqin Li},
  title = {{CCDE: A Compact and Competitive Dialogue Evaluation Framework via Knowledge Distillation of Large Language Models}},
  journal = {IEEE Transactions on Computational Social Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=doCAoubEGN},
}

Loading related content...

Figure