RECKON: Large-scale Reference-based Efficient Knowledge Evaluation for Large Language Model

Lin Zhang, Zhouhong Gu, Xiaoran Shi, Hongwei Feng, Yanghua XiaoarXiv 2025 · DOI · arXiv

Introduction

As Large Language Models (LLMs) continue to advance, the methods used to evaluate their knowledge and capabilities must also evolve. Traditional evaluation relies heavily on static benchmarks, such as MMLU, which consist of fixed multiple-choice or fill-in-the-blank questions. However, these benchmarks are expensive to develop, quickly become obsolete as models are trained on newer data, and often suffer from 'information loss' when complex real-world information is reduced to rigid formats.

An alternative approach, 'Judge with Reference' (Jw/R), involves providing a judge model with full reference documents to evaluate a target model's response. While highly accurate, Jw/R is prohibitively expensive due to the massive token consumption required to process entire documents. To bridge this gap, we introduce RECKON (Reference-based Efficient Knowledge Evaluation), a framework that maintains the accuracy of reference-based evaluation while significantly reducing costs through dynamic question generation and semantic clustering.

The RECKON Framework

RECKON bypasses the limitations of static benchmarks by generating questions directly from unstructured reference data. The framework employs a five-step iterative process to ensure comprehensive and efficient evaluation.

The five-step iterative workflow of the RECKON framework.
  1. Preparing Knowledge Units: Unstructured reference data is decomposed into discrete, manageable facts or concepts known as 'knowledge units.'
  2. Clustering: These units are converted into embeddings and grouped into thematic clusters based on semantic similarity, allowing for organized evaluation.
  3. Question Generation: An LLM generates targeted, contextually appropriate questions designed to cover the specific knowledge units within a cluster.
  4. Get Response: The target LLM being evaluated provides answers to these generated questions.
  5. Judging: An evaluator LLM checks the target model's response against the required knowledge units. Any units that are missed or incorrectly addressed are recycled into subsequent rounds for re-evaluation.

This dynamic loop ensures that the evaluation is thorough without the redundancy of feeding entire documents into every prompt, leading to a more streamlined process.

Experimental Setup and Datasets

The effectiveness of RECKON was tested across several diverse textual corpora:

  • Wikitext (Wiki): For general world knowledge.
  • CodeGPT: For programming and technical assignments.
  • Legalbench: For juridical definitions and legal reasoning.
  • PubMedQA (Med.rand & Med.sim): For biomedical knowledge and entity relationships.
  • Feverous (Uncommon): A dataset with descriptions that deviate from established facts, used to test if models rely on internal biases rather than the provided reference.

Analysis of Model Performance

Question Type Sensitivity

One of the key insights from our evaluation is that LLM performance is not uniform across different types of queries. Models consistently underperform on identity-based ('who') and location-based ('where') questions compared to general factual recall ('what').

Model performance categorized by question type (what, when, where, who, why, how).

Information Density and Bias

We observed that LLM performance degrades when processing knowledge units with high information density, such as long lists of entities. RECKON's clustering approach helps mitigate this by organizing data into digestible chunks. Additionally, grounding evaluations in specific reference units helps neutralize 'internal model bias,' where a judge model might incorrectly mark an answer as correct because it aligns with its own flawed internal knowledge.

Key Results

RECKON achieves a remarkable balance between efficiency and accuracy:

  • Cost Efficiency: The framework reduces resource consumption by 56.5% compared to standard Jw/R methods. In a test using the Wiki dataset, evaluating GPT-3.5 via Jw/R cost $38.14, while RECKON achieved the same result for only **$2.44**.
  • High Accuracy: RECKON shows a 97.6% to 99.5% correlation with human annotators, significantly outperforming traditional metrics like BLEU (28-38%) and GPTScore (41-62%).
  • Model Benchmarking: GPT-4 emerged as the top performer (e.g., 97.9% on Med.sim), though domain-specific models like PMC-LLaMA showed superior performance in specialized fields like biomedicine compared to general-purpose models.

Conclusion

RECKON provides a scalable and cost-effective solution for the continuous evaluation of LLMs. By leveraging dynamic question generation and semantic clustering, it offers a high-fidelity alternative to static benchmarks. Future work will focus on refining question generation to reduce ambiguity and establishing ethical protocols for handling sensitive reference data.

Summary updated

Abstract

As large language models (LLMs) advance, efficient knowledge evaluation becomes crucial to verifying their capabilities. Traditional methods, relying on benchmarks, face limitations such as high resource costs and information loss. We propose the Large-scale Reference-based Efficient Knowledge Evaluation for Large Language Model (RECKON), which directly uses reference data to evaluate models. RECKON organizes unstructured data into manageable units and generates targeted questions for each cluster, improving evaluation accuracy and efficiency. Experimental results show that RECKON reduces resource consumption by 56.5% compared to traditional methods while achieving over 97% accuracy across various domains, including world knowledge, code, legal, and biomedical datasets. Code is available at https://github.com/MikeGu721/reckon
BibTeX
@misc{zhang2025reckon,
  author = {Lin Zhang and Zhouhong Gu and Xiaoran Shi and Hongwei Feng and Yanghua Xiao},
  title = {{RECKON: Large-scale Reference-based Efficient Knowledge Evaluation for Large Language Model}},
  year = {2025},
  eprint = {2504.00756},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.00756},
}
Figure