Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores

Robert E. Blackwell, Jon Barry, Anthony G. Cohn

TL;DR

Reproducibility in LLM evaluations is hampered by their inherent stochasticity. This paper proposes a practical method for quantifying this uncertainty by analyzing the spread of benchmark scores across multiple runs, suggesting that setting temperature to 0 and using a fixed seed significantly reduces variability and the number of repeats needed for stable results.

What Problem Does This Paper Address?

As Large Language Models (LLMs) become increasingly sophisticated and widely adopted, assessing their performance through benchmarks is crucial. However, LLMs are inherently stochastic, meaning they don't always produce the same output even for identical inputs. This variability, stemming from probabilistic sampling mechanisms and distributed system complexities, makes benchmark scores unstable and difficult to reproduce. Existing evaluation practices often overlook this uncertainty, potentially leading to misleading comparisons and a lack of confidence in reported results. The cost and time associated with extensive repeated experiments further discourage thorough uncertainty quantification.

Methods

The authors investigated the impact of experimental repeats on benchmark scores and their associated uncertainty. They employed two qualitative spatial reasoning benchmarks: "Small" (100 questions) and "Large" (5760 questions), both focused on cardinal directions. They tested six representative LLMs (GPT-3.5T, GPT-4o, Llama-3 70B, Llama-3 7B, Gemini-1.5P, and Claude-3.5S) across different configurations: default settings and a controlled setting with temperature set to 0 and a fixed random seed (where applicable).

The core of their method involves calculating a prediction interval for future observations of the mean score. This interval, derived from the standard deviation of scores across multiple repeats, provides a quantitative measure of uncertainty. They systematically varied the number of experimental repeats (from 1 to 30) to observe how the prediction interval width converges, aiming for a width below a desirable threshold (e.g., 0.01). They also conducted a two-sample t-test to compare results from different API providers for the same model.

Datasets

The paper utilizes two existing spatial reasoning benchmarks:

  • Small Benchmark: Consists of 100 simple questions requiring cardinal direction answers (e.g., "You are watching the sun set. Which direction are you facing?").
  • Large Benchmark: Comprises 5760 templated questions with cardinal or intercardinal direction answers (e.g., "You are walking south along the east shore of a lake; in which direction is the lake?").

No new benchmarks were introduced in this study.

Key Insights and Ideas

  • Stochasticity is Pervasive: Even with temperature set to zero and a fixed seed, perfect determinism isn't guaranteed across all LLMs and benchmarks, though variability is significantly reduced.
  • Temperature and Seed Matter: Setting temperature to 0 and using a fixed seed (when available) is a highly effective strategy for reducing answer variability and achieving stable benchmark scores with fewer repeats.
  • Benchmark Size Impacts Convergence: Larger benchmarks with more questions generally lead to faster convergence of the prediction interval width, meaning fewer repeats are needed to achieve a desired level of certainty.
  • API Hosting Can Matter: Differences in how LLMs are hosted (e.g., Azure OpenAI API vs. OpenAI API) can lead to statistically significant differences in benchmark scores, even for the same model version.
  • Cost-Effective Uncertainty Quantification: The proposed method of using prediction intervals with a limited number of repeats (often as few as 3-4) offers a practical and cost-effective way to quantify LLM evaluation uncertainty.

Key Results

  • For most models and benchmarks, setting temperature to 0 and using a fixed seed reduced the prediction interval width to below 0.01 within 3-4 repeats.
  • The "Large" benchmark generally required fewer repeats to achieve a prediction interval width below 0.01 compared to the "Small" benchmark.
  • GPT-3.5T scores from the Azure OpenAI API and the OpenAI API on the "Small" benchmark were statistically significantly different (t = 2.51, p = 0.013), suggesting potential differences due to hosting or configuration.
  • Llama-3 7B consistently produced deterministic results when run locally, likely due to the absence of distributed parallelism challenges.

Limitations and Future Work

Limitations:

  • The study tested a limited number of LLMs (six) and benchmarks (two). Many other LLMs with different architectures, weights, and parameters exist, and other benchmarks might require more repeats for stable results.
  • The impact of prompt engineering on answer variability was not investigated.
  • Nucleus (top-p) sampling was not explored, though preliminary work suggests it might also help reduce variability.
  • The study used frequentist statistics; exploring Bayesian approaches for uncertainty quantification could be beneficial.
  • LLMs are black boxes, and underlying hardware/software systems can change, impacting reproducibility.

Future Work:

  • Investigate the impact of prompt engineering and linguistic variations on uncertainty.
  • Explore the effectiveness of nucleus sampling for reducing variability.
  • Apply Bayesian statistical approaches to uncertainty quantification.
  • Conduct similar analyses on a wider range of LLMs and benchmarks.
  • Develop standardized reporting practices for LLM evaluation that include uncertainty measures.

Summary updated

Abstract

Large language models (LLMs) are stochastic, and not all models give deterministic answers, even when setting temperature to zero with a fixed random seed. However, few benchmark studies attempt to quantify uncertainty, partly due to the time and cost of repeated experiments. We use benchmarks designed for testing LLMs' capacity to reason about cardinal directions to explore the impact of experimental repeats on mean score and prediction interval. We suggest a simple method for cost-effectively quantifying the uncertainty of a benchmark score and make recommendations concerning reproducible LLM evaluation.
BibTeX
@misc{blackwell2024towards,
  author = {Robert E. Blackwell and Jon Barry and Anthony G. Cohn},
  title = {{Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores}},
  year = {2024},
  eprint = {2410.03492},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.03492},
}
Figure