Abstract
Evaluating Large Language Models (LLMs) typically requires either costly human intervention or reliance on proprietary, closed-source models like GPT-4. We present Fennec, a framework that empowers smaller, 7B-parameter open-source models to conduct high-quality, fine-grained evaluation and response correction. Fennec employs two primary techniques: "branching," which structures evaluation into a multi-step reasoning process, and "bridging," which synthesizes diverse datasets into a unified training corpus. Our results demonstrate that Fennec-7B achieves performance levels competitive with GPT-4 and significantly outperforms larger open-source evaluators.
Introduction
As Large Language Models (LLMs) are increasingly deployed in complex real-world scenarios, the need for reliable, scalable evaluation becomes critical. While human evaluation remains the gold standard, its cost and latency are prohibitive. The industry has shifted toward "LLM-as-a-judge" approaches, often relying on proprietary models. However, these present challenges regarding cost, transparency, and version stability. Furthermore, existing open-source evaluators often function as "black boxes," providing scores without actionable feedback. Fennec addresses these gaps by providing a transparent, open-source evaluator that offers both multi-dimensional grading and specific corrections.
Methodology
The Fennec framework is built upon two core operations designed to maximize the reasoning capabilities of smaller models.
Branching
Inspired by Chain-of-Thought (CoT) prompting, the branching operation decomposes the evaluation task into four sequential steps:
- Constructing Evaluation Criteria: Identifying specific dimensions like accuracy, coherence, and conciseness.
- Specifying Scoring Guidelines: Defining the qualitative difference between score levels (e.g., 1 to 5).
- Rendering Judgments: Providing a score accompanied by a detailed explanation.
- Conducting Corrections: Generating a refined version of the original response based on the evaluation.
Bridging
To overcome the scarcity of fine-grained training data, the authors developed a "bridging" technique. Many legacy datasets contain only prompts and final scores. Fennec utilizes a reverse model to reconstruct the missing intermediate reasoning steps (criteria and guidelines) for these datasets. This allows for the integration of diverse sources like Auto-J, JudgeLM, and Prometheus into a unified, high-quality training corpus.
Experimental Results
Fennec-7B was evaluated against several benchmarks, including Auto-P, PandaLM-test, and MT-Bench.
Performance and Consistency
The results indicate that Fennec-7B "punches above its weight," often outperforming 13B parameter models and rivaling GPT-4. Notably, on the PandaLM-test, Fennec achieved a consistency rate of 89.69%. As shown in the performance comparison, the "bridging" operation significantly enhances the model's agreement and consistency across all tested benchmarks.
Response Correction
Beyond evaluation, Fennec serves as an effective refiner. By applying Fennec's corrections, the MT-Bench scores of existing models improved significantly. For instance, LLaMA2-7B saw a 0.89-point increase, while Alpaca-13B improved by 1.87 points. This demonstrates that fine-grained evaluation naturally leads to high-quality model improvement.
Discussion and Conclusion
Fennec proves that structured reasoning ("branching") can compensate for smaller model scales. While the multi-step generation process introduces higher inference latency compared to score-only models, the gain in transparency and the ability to provide automated corrections offer significant value. Future work will focus on optimizing inference speed and extending the framework to multi-turn dialogues and more specialized knowledge domains.
Summary updated