Introduction
As Large Language Models (LLMs) continue to advance, the methods used to evaluate their knowledge and capabilities must also evolve. Traditional evaluation relies heavily on static benchmarks, such as MMLU, which consist of fixed multiple-choice or fill-in-the-blank questions. However, these benchmarks are expensive to develop, quickly become obsolete as models are trained on newer data, and often suffer from 'information loss' when complex real-world information is reduced to rigid formats.
An alternative approach, 'Judge with Reference' (Jw/R), involves providing a judge model with full reference documents to evaluate a target model's response. While highly accurate, Jw/R is prohibitively expensive due to the massive token consumption required to process entire documents. To bridge this gap, we introduce RECKON (Reference-based Efficient Knowledge Evaluation), a framework that maintains the accuracy of reference-based evaluation while significantly reducing costs through dynamic question generation and semantic clustering.
The RECKON Framework
RECKON bypasses the limitations of static benchmarks by generating questions directly from unstructured reference data. The framework employs a five-step iterative process to ensure comprehensive and efficient evaluation.
- Preparing Knowledge Units: Unstructured reference data is decomposed into discrete, manageable facts or concepts known as 'knowledge units.'
- Clustering: These units are converted into embeddings and grouped into thematic clusters based on semantic similarity, allowing for organized evaluation.
- Question Generation: An LLM generates targeted, contextually appropriate questions designed to cover the specific knowledge units within a cluster.
- Get Response: The target LLM being evaluated provides answers to these generated questions.
- Judging: An evaluator LLM checks the target model's response against the required knowledge units. Any units that are missed or incorrectly addressed are recycled into subsequent rounds for re-evaluation.
This dynamic loop ensures that the evaluation is thorough without the redundancy of feeding entire documents into every prompt, leading to a more streamlined process.
Experimental Setup and Datasets
The effectiveness of RECKON was tested across several diverse textual corpora:
- Wikitext (Wiki): For general world knowledge.
- CodeGPT: For programming and technical assignments.
- Legalbench: For juridical definitions and legal reasoning.
- PubMedQA (Med.rand & Med.sim): For biomedical knowledge and entity relationships.
- Feverous (Uncommon): A dataset with descriptions that deviate from established facts, used to test if models rely on internal biases rather than the provided reference.
Analysis of Model Performance
Question Type Sensitivity
One of the key insights from our evaluation is that LLM performance is not uniform across different types of queries. Models consistently underperform on identity-based ('who') and location-based ('where') questions compared to general factual recall ('what').
Information Density and Bias
We observed that LLM performance degrades when processing knowledge units with high information density, such as long lists of entities. RECKON's clustering approach helps mitigate this by organizing data into digestible chunks. Additionally, grounding evaluations in specific reference units helps neutralize 'internal model bias,' where a judge model might incorrectly mark an answer as correct because it aligns with its own flawed internal knowledge.
Key Results
RECKON achieves a remarkable balance between efficiency and accuracy:
- Cost Efficiency: The framework reduces resource consumption by 56.5% compared to standard Jw/R methods. In a test using the Wiki dataset, evaluating GPT-3.5 via Jw/R cost $38.14, while RECKON achieved the same result for only **$2.44**.
- High Accuracy: RECKON shows a 97.6% to 99.5% correlation with human annotators, significantly outperforming traditional metrics like BLEU (28-38%) and GPTScore (41-62%).
- Model Benchmarking: GPT-4 emerged as the top performer (e.g., 97.9% on Med.sim), though domain-specific models like PMC-LLaMA showed superior performance in specialized fields like biomedicine compared to general-purpose models.
Conclusion
RECKON provides a scalable and cost-effective solution for the continuous evaluation of LLMs. By leveraging dynamic question generation and semantic clustering, it offers a high-fidelity alternative to static benchmarks. Future work will focus on refining question generation to reduce ambiguity and establishing ethical protocols for handling sensitive reference data.
Summary updated