A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations

Md. Tahmid Rahman Laskar, Sawsan Alqahtani, M. Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee-Wei Tan, Md. Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang

Abstract

As Large Language Models (LLMs) transition from experimental research to production-grade applications, the methodologies used to evaluate their performance have come under intense scrutiny. This paper provides a systematic survey and critical review of the current LLM evaluation landscape, identifying a "crisis of evaluation" characterized by inconsistent setups, poor documentation, and brittle metrics. By categorizing these failures into issues of reproducibility, reliability, and robustness, the authors propose a roadmap for more scientific and standardized benchmarking.

Introduction: The "Wild West" of LLM Evaluation

The rapid proliferation of Large Language Models has outpaced the development of robust evaluation frameworks. Currently, the industry faces a significant challenge: conflicting reports where a specific model may dominate one leaderboard while underperforming on another. This inconsistency stems from a lack of standardization across the evaluation pipeline. Small, often undocumented variables—such as specific prompt phrasing, decoding temperatures, or the specific version of a benchmark—can lead to vastly different performance metrics. This paper addresses these systemic issues, advocating for a transition from anecdotal reporting to rigorous, scientific evaluation.

Methodology

This study functions as a systematic survey and meta-analysis, synthesizing findings from hundreds of recent research papers. The authors' approach is three-fold:

  1. Typology Development: The LLM evaluation workflow is decomposed into three critical stages: Evaluation Setup, Response Generation, and Evaluation Methodology.
  2. Meta-Analysis: An analysis of 212 recent papers was conducted to quantify the "transparency gap" in the field.
  3. Targeted Case Studies: The authors performed controlled experiments to demonstrate the sensitivity of state-of-the-art models (e.g., GPT-4o, Claude-3) to minor prompt variations and tokenizer differences.

The Evolution of Benchmarking Datasets

The paper reviews a broad spectrum of benchmarks, categorizing them into general capabilities (MMLU, ARC), specialized domains like coding or math (HumanEval, GSM-8K), and composite suites (HELM, MixEval). A key contribution of this review is the advocacy for "Refined Benchmarks." Traditional benchmarks often suffer from data contamination, where test questions are inadvertently included in the model's training data, or contain outdated "gold" labels. The authors recommend a shift toward improved versions such as MMLU-Pro, HumanEvalPlus, and GSM-1K to restore the credibility of model rankings.

Key Insights and Systemic Failures

The Prompting Paradox

One of the most significant findings is the extreme brittleness of LLM performance relative to prompting. For instance, adding a simple length constraint can improve performance in one model while degrading it in another. This "Prompting Paradox" implies that without the release of exact prompt strings, research results are fundamentally irreproducible.

The "Moving Target" Problem

The reliance on closed-source models (e.g., GPT-4) as baselines creates a temporal instability in research. Because these models are updated frequently without public versioning logs, a score recorded six months ago may no longer be a valid point of comparison for the "same" model today.

Tokenization and Vocabulary Mismatch

The authors identify that tokenizers often cover only a small fraction of a benchmark's vocabulary. Consequently, a model might fail a test not due to a lack of reasoning capability, but because of how it perceives the input text at the sub-word level.

The Failure of Automated Metrics

Traditional metrics like ROUGE or Exact Match are increasingly inadequate for the generative era. These metrics often penalize synonymous but correct answers. The paper demonstrates that hybrid evaluation—combining automated parsing with human-in-the-loop verification—can correct error rates by more than 10%.

Empirical Results

The meta-analysis revealed a startling lack of transparency in the field:

  • Transparency Gap: Only 29.3% of the 212 papers reviewed reported the specific version of the model used.
  • Fairness in Comparison: Only 20.7% of model comparisons were deemed "fair," meaning they utilized identical subsets and settings.
  • Prompt Sensitivity: In experiments using the SAMSum dataset, "restricted" prompting styles consistently outperformed "unrestricted" styles across GPT-4o and Claude-3-Opus, suggesting that many leaderboards may be measuring prompt engineering skills rather than underlying intelligence.

Limitations and Future Directions

The authors acknowledge that their review is primarily focused on English-language, text-only benchmarks. Future work is required to address the unique challenges of multilingual evaluation and the rapidly evolving multimodal (vision+text) landscape. Furthermore, the study focuses on the evaluation phase, leaving the complexities of training and fine-tuning for separate investigation.

To move forward, the paper calls for the development of standardized, version-controlled evaluation protocols and the adoption of "dynamic" benchmarks that can evolve to prevent data contamination. Additionally, the authors suggest using "juries" of multiple LLMs to mitigate the biases inherent in "LLM-as-a-judge" approaches.

Conclusion

LLM evaluation is currently in a state of crisis, but by identifying the specific points of failure in reproducibility, reliability, and robustness, this survey provides a necessary roadmap for the community. Moving toward standardized, transparent, and refined benchmarking is essential for the continued healthy development of artificial intelligence.

Summary updated

Abstract

Large Language Models (LLMs) have recently gained significant attention due to their remarkable capabilities in performing diverse tasks across various domains. However, a thorough evaluation of these models is crucial before deploying them in real-world applications to ensure they produce reliable performance. Despite the well-established importance of evaluating LLMs in the community, the complexity of the evaluation process has led to varied evaluation setups, causing inconsistencies in findings and interpretations. To address this, we systematically review the primary challenges and limitations causing these inconsistencies and unreliable evaluations in various steps of LLM evaluation. Based on our critical review, we present our perspectives and recommendations to ensure LLM evaluations are reproducible, reliable, and robust.
BibTeX
@inproceedings{laskar2024systematic,
  author = {Md. Tahmid Rahman Laskar and Sawsan Alqahtani and M. Saiful Bari and Mizanur Rahman and Mohammad Abdullah Matin Khan and Haidar Khan and Israt Jahan and Amran Bhuiyan and Chee-Wei Tan and Md. Rizwan Parvez and Enamul Hoque and Shafiq Joty and Jimmy Huang},
  title = {{A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.764},
}
Figure