Fennec: Fine-grained Language Model Evaluation and Correction Extended through Branching and Bridging

Xiaobo Liang, Haoke Zhang, Helan hu, Juntao Li, Jun Xu, Min ZhangarXiv 2024 · DOI · arXiv

Abstract

Evaluating Large Language Models (LLMs) typically requires either costly human intervention or reliance on proprietary, closed-source models like GPT-4. We present Fennec, a framework that empowers smaller, 7B-parameter open-source models to conduct high-quality, fine-grained evaluation and response correction. Fennec employs two primary techniques: "branching," which structures evaluation into a multi-step reasoning process, and "bridging," which synthesizes diverse datasets into a unified training corpus. Our results demonstrate that Fennec-7B achieves performance levels competitive with GPT-4 and significantly outperforms larger open-source evaluators.

Introduction

As Large Language Models (LLMs) are increasingly deployed in complex real-world scenarios, the need for reliable, scalable evaluation becomes critical. While human evaluation remains the gold standard, its cost and latency are prohibitive. The industry has shifted toward "LLM-as-a-judge" approaches, often relying on proprietary models. However, these present challenges regarding cost, transparency, and version stability. Furthermore, existing open-source evaluators often function as "black boxes," providing scores without actionable feedback. Fennec addresses these gaps by providing a transparent, open-source evaluator that offers both multi-dimensional grading and specific corrections.

Methodology

The Fennec framework is built upon two core operations designed to maximize the reasoning capabilities of smaller models.

Branching

Inspired by Chain-of-Thought (CoT) prompting, the branching operation decomposes the evaluation task into four sequential steps:

  1. Constructing Evaluation Criteria: Identifying specific dimensions like accuracy, coherence, and conciseness.
  2. Specifying Scoring Guidelines: Defining the qualitative difference between score levels (e.g., 1 to 5).
  3. Rendering Judgments: Providing a score accompanied by a detailed explanation.
  4. Conducting Corrections: Generating a refined version of the original response based on the evaluation.
The Fennec branching workflow deconstructs evaluation into criteria, guidelines, judgment, and correction.

Bridging

To overcome the scarcity of fine-grained training data, the authors developed a "bridging" technique. Many legacy datasets contain only prompts and final scores. Fennec utilizes a reverse model to reconstruct the missing intermediate reasoning steps (criteria and guidelines) for these datasets. This allows for the integration of diverse sources like Auto-J, JudgeLM, and Prometheus into a unified, high-quality training corpus.

Experimental Results

Fennec-7B was evaluated against several benchmarks, including Auto-P, PandaLM-test, and MT-Bench.

Performance and Consistency

The results indicate that Fennec-7B "punches above its weight," often outperforming 13B parameter models and rivaling GPT-4. Notably, on the PandaLM-test, Fennec achieved a consistency rate of 89.69%. As shown in the performance comparison, the "bridging" operation significantly enhances the model's agreement and consistency across all tested benchmarks.

Performance comparison of Fennec against open-source and closed-source models across multiple benchmarks.

Response Correction

Beyond evaluation, Fennec serves as an effective refiner. By applying Fennec's corrections, the MT-Bench scores of existing models improved significantly. For instance, LLaMA2-7B saw a 0.89-point increase, while Alpaca-13B improved by 1.87 points. This demonstrates that fine-grained evaluation naturally leads to high-quality model improvement.

Discussion and Conclusion

Fennec proves that structured reasoning ("branching") can compensate for smaller model scales. While the multi-step generation process introduces higher inference latency compared to score-only models, the gain in transparency and the ability to provide automated corrections offer significant value. Future work will focus on optimizing inference speed and extending the framework to multi-turn dialogues and more specialized knowledge domains.

Summary updated

Abstract

The rapid advancement of large language models has given rise to a plethora of applications across a myriad of real-world tasks, mainly centered on aligning with human intent. However, the complexities inherent in human intent necessitate a dependence on labor-intensive and time-consuming human evaluation. To alleviate this constraint, we delve into the paradigm of employing open-source large language models as evaluators, aligning with the prevailing trend of utilizing GPT-4. Particularly, we present a step-by-step evaluation framework: \textbf{Fennec}, capable of \textbf{F}ine-grained \textbf{E}valuatio\textbf{N} and correctio\textbf{N} \textbf{E}xtended through bran\textbf{C}hing and bridging. Specifically, the branching operation dissects the evaluation task into various dimensions and granularities, thereby alleviating the challenges associated with evaluation. Concurrently, the bridging operation amalgamates diverse training datasets, augmenting the variety of evaluation tasks. In experimental trials, our 7B model consistently outperforms open-source larger-scale evaluation models across various widely adopted benchmarks in terms of both \textit{Agreement} and \textit{Consistency}, closely approaching the capabilities of GPT-4. We employ the fine-grained correction capabilities induced by the evaluation model to refine multiple model responses, and the results show that the refinement elevates the quality of responses, leading to an improvement of 1-2 points on the MT-Bench. Our code is available at Github\footnote{\url{https://github.com/dropreg/Fennec}}.
BibTeX
@misc{liang2024fennec,
  author = {Xiaobo Liang and Haoke Zhang and Helan hu and Juntao Li and Jun Xu and Min Zhang},
  title = {{Fennec: Fine-grained Language Model Evaluation and Correction Extended through Branching and Bridging}},
  year = {2024},
  eprint = {2405.12163},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.12163},
}
Figure