A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations

Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang

TL;DR

This paper provides a comprehensive "health check" for the current state of Large Language Model (LLM) evaluation. By breaking down the evaluation process into a three-step workflow, the authors identify critical failures in reproducibility, reliability, and robustness, ultimately offering a roadmap of best practices to move beyond the "wild west" of inconsistent benchmarking.

What Problem Does This Paper Address?

We are currently in an arms race of LLM releases, where every new model claims "state-of-the-art" performance on standard benchmarks. However, the methods used to reach these numbers are often a black box. The authors argue that the community suffers from a "reproducibility crisis" in evaluation: researchers often fail to release their exact prompts, decoding parameters, or the specific data subsets used.

Furthermore, existing benchmarks are plagued by data contamination (where models are trained on the test questions) and inaccurate gold labels. This makes it nearly impossible to tell if a model is truly "smarter" or just better at memorizing the test. The paper seeks to standardize how we measure intelligence in these systems to ensure they are safe and reliable for real-world deployment.

Methods

The authors conduct a systematic survey and critical review, proposing a standardized LLM Evaluation Workflow (Figure 1) categorized into three main stages:

  1. Evaluation Setup: Selecting benchmarks and models.
  2. Response Generation: Designing prompts and configuring decoding parameters (like temperature and top-p).
  3. Evaluation Methodology: Designing parsing scripts to extract answers and choosing the right metrics (automatic vs. human).
Figure 1: Typology of the LLM Evaluation Workflow. A more detailed description of the challenges and the limitations can be found in Table 5.

They analyze 212 papers to quantify resource availability and perform small-scale experiments to demonstrate how sensitive models are to minor changes in prompt design and decoding settings.

Datasets

The paper reviews a wide array of benchmarks, categorizing them into:

  • General Capability: MMLU, HumanEval, GSM8K, and HellaSwag.
  • Specialized: MT-Bench (chat), RewardBench (reward models), and Open Medical-LLM.
  • Diverse/Aggregated: HELM and MixEval.

Crucially, the authors highlight the shift toward "Plus" or "Pro" versions of datasets (e.g., MMLU-Pro, HumanEvalPlus, GSM-1K) designed to be more robust against contamination and to provide more rigorous testing than their predecessors.

Key Insights and Ideas

  • The Prompt Sensitivity Trap: Minor variations in how a prompt is structured can lead to massive swings in performance. As shown in Figure 4, "restricted" vs. "unrestricted" prompting can completely flip the ranking of top-tier models like GPT-4o and Claude-3-Opus. Impact of prompt tuning on ROUGE-1 scores across different models, showing significant performance variance.

  • Vocabulary Coverage Gap: The authors found that popular benchmarks only exercise a tiny fraction of a model's vocabulary (often less than 1% for MixEval-Hard), suggesting that these tests don't truly capture the "generalized" intelligence of the models.

  • LLM-as-a-Judge Biases: While using LLMs to grade other LLMs is efficient, it introduces "self-preference" bias (models favoring their own style) and positional bias. The authors recommend using "juries" of multiple models to mitigate this.
  • Automatic vs. Hybrid Evaluation: Purely automated parsing of LLM responses can be misleading. In open-domain QA, human intervention (Hybrid Evaluation) often reveals that models generated correct, synonymous answers that the automated script marked as wrong (Figure 2). Comparison of Automatic vs. Hybrid evaluation scores, showing how human intervention corrects for outdated or synonymous gold labels.
Figure 4: ROUGE-1 scores in the SAMSum dataset based on Prompt Tuning.
Figure 2: Comparing Automatic and Hybrid Evalua- tion.

Key Results

The paper demonstrates that model rankings are highly unstable. For example, when comparing LLaMA-3-8B and Qwen2-7B across different benchmarks, the "winner" changes depending on the specific task (Figure 3). Qwen2 dominates in HumanEval (coding), but LLaMA-3 takes the lead in GPQA (science).

Figure 3: Performance Comparison: LLaMA-3 and Qwen2

Furthermore, the authors show a disconnect between academic benchmarks and human preference. Table 3 (Figure 6) reveals that a model's rank on MMLU (a multiple-choice knowledge test) often disagrees with its rank on the LMSys Chatbot Arena (a human-voted leaderboard).

Rankings of models on LMSys Chatbot Arena vs. MMLU implementations, showing significant disagreement between metrics.

Limitations and Future Work

  • Monolingual Focus: The survey primarily focuses on English-language evaluations, leaving the challenges of multilingual LLM evaluation for future study.
  • Text-Only: Multimodal capabilities (vision + language) are acknowledged but not analyzed in depth.
  • Closed-Source Barriers: The authors note that without access to model weights, it is mathematically impossible to fully prevent data contamination or verify if a model has "seen" the test data during training.
  • Future Direction: The authors call for the development of standardized, dynamic evaluation protocols that evolve as models get smarter, moving away from static, easily-leaked test sets.

Summary updated

Abstract

Large Language Models (LLMs) have recently gained significant attention due to their remarkable capabilities in performing diverse tasks across various domains. However, a thorough evaluation of these models is crucial before deploying them in real-world applications to ensure they produce reliable performance. Despite the well-established importance of evaluating LLMs in the community, the complexity of the evaluation process has led to varied evaluation setups, causing inconsistencies in findings and interpretations. To address this, we systematically review the primary challenges and limitations causing these inconsistencies and unreliable evaluations in various steps of LLM evaluation. Based on our critical review, we present our perspectives and recommendations to ensure LLM evaluations are reproducible, reliable, and robust.
BibTeX
@inproceedings{laskar2024systematic,
  author = {Md Tahmid Rahman Laskar and Sawsan Alqahtani and M Saiful Bari and Mizanur Rahman and Mohammad Abdullah Matin Khan and Haidar Khan and Israt Jahan and Amran Bhuiyan and Chee Wei Tan and Md Rizwan Parvez and Enamul Hoque and Shafiq Joty and Jimmy Huang},
  title = {{A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.764},
}
Figure