Evaluation of human-model prediction difference on the Internet Scale of Data

Weitang Liu, Ying Wai Li, Tianle Wang, Yi‐Zhuang You, Jingbo Shang, Shang, Jingbo

Abstract

Evaluating models on datasets often fails to capture their behavior when faced with unexpected and diverse types of inputs. It would be beneficial if we could evaluate the difference between human annotation and model prediction for an internet number of inputs, or more generally, for an input space that enumeration is computationally impractical. Traditional model evaluation methods rely on precision and recall (PR) as metrics, which are typically estimated by comparing human annotations with model predictions on a specific dataset. This is feasible because enumerating thousands of test inputs is manageable. However, estimating PR across a large input space is challenging because enumeration becomes computationally infeasible. We propose OmniInput, a novel approach to evaluate and compare NNs by the PR of an input space. OmniInput is distinctive from previous works as its estimated PR reflects the estimation of the differences between human annotation and model prediction in the input space which is usually too huge to be enumerated. We empirically validate our method within an enumerable input space, and our experiments demonstrate that OmniInput can effectively estimate and compare precision and recall for (large) language models within a broad input space that is not enumerable.

Introduction

Standard AI evaluation often resembles testing a vehicle only on a pristine, indoor track. While it provides a baseline for performance in controlled environments, it fails to predict how a model will handle the chaotic, "internet-scale" variety of real-world inputs. In the domain of Large Language Models (LLMs), current benchmarks like MMLU or GSM8K are static and finite. Once deployed, these models encounter a near-infinite space of possible text sequences that no fixed dataset can fully represent.

Traditional evaluation relies on Precision and Recall (PR) calculated on specific, finite datasets. This approach faces two major hurdles at scale: enumeration is computationally impossible for long sequences, and uniform sampling is inefficient, as random strings are mostly gibberish. Models typically only predict with high confidence on a tiny sliver of the total input space. To address this, we introduce OMNIINPUT, a framework designed to estimate PR metrics across vast, non-enumerable input spaces by leveraging the model’s internal probability distribution.

Methodology

OMNIINPUT uses the model itself to map the input space through a four-step statistical process:

  1. Sampling the Output Distribution: Using an efficient sampler (Parallel Tempering and Histogram Reweighting), we obtain the distribution $\rho(z)$ of the model's outputs based on Negative Log-Likelihood (NLL). This identifies how many potential inputs fall into specific "confidence" bins.
  2. Targeted Annotation: Instead of labeling random data, human annotators evaluate a representative set of samples from each NLL bin, rating the "human-likeness" or correctness of the model's prediction on a scale of 0 to 1.
  3. Precision per Bin: The average human score for a bin is defined as the "precision per bin," $r(z)$.
  4. Global PR Estimation: By weighting the precision of each bin by the total number of inputs the model assigns to that bin, the framework reconstructs global PR curves for the entire innumerable input space.
The OMNIINPUT workflow: from sampling NLL bins to generating a global PR curve.

Empirical Validation

To verify the accuracy of OMNIINPUT, we first tested it in an enumerable environment using a toy GPT-2 model trained on simple mathematical rules. With an input space of $10^8$ sequences, we could calculate the ground truth via full enumeration. As shown in the validation results, the sampled estimates for both the distribution $\rho(z)$ and the precision $r(z)$ closely align with the ground truth, confirming the statistical reliability of the framework.

Validation results comparing sampled estimates against ground truth enumeration for precision r(z) and distribution rho(z).

Key Insights and Results

Our experiments across pre-trained LLMs (GPT-2, Llama-1, Llama-2) and classifiers (DistilBERT) yielded several critical insights into model behavior in the open-world space:

  • The "Low NLL" Trap: While lower NLL (higher confidence) usually correlates with better quality, OMNIINPUT revealed a drop in precision at extremely low NLL values. In these regions, models often fall into repetitive, nonsensical loops that humans rate poorly.
  • Uncovering Hidden Behaviors: By exploring the model's internal probability landscape rather than a fixed dataset, the sampler identified "corners" of the model's knowledge, revealing privacy leaks (e.g., generation of real-looking addresses) and hallucinations that standard benchmarks miss.
  • Model Comparison: In testing 25-token sequences, Llama-1 demonstrated higher precision than Llama-2 within the specific NLL ranges explored. Furthermore, we observed a significant "length penalty"; as sequence length increased from 25 to 100 tokens, the precision of GPT-2 models dropped sharply, indicating a struggle to maintain logic over longer spans in unconstrained spaces.

Conclusion

OMNIINPUT provides a scalable alternative to static benchmarks by estimating model performance across nearly infinite input spaces. By combining efficient MCMC sampling with targeted human annotation, it uncovers qualitative failures—such as privacy risks and repetitive loops—that remain hidden in traditional evaluations. Future work will focus on scaling these samplers for frontier models and exploring automated annotation to further reduce the human-in-the-loop cost.

Summary updated

Abstract

Evaluating models on datasets often fails to capture their behavior when faced with unexpected and diverse types of inputs. It would be beneficial if we could evaluate the difference between human annotation and model prediction for an internet number of inputs, or more generally, for an input space that enumeration is computationally impractical. Traditional model evaluation methods rely on precision and recall (PR) as metrics, which are typically estimated by comparing human annotations with model predictions on a specific dataset. This is feasible because enumerating thousands of test inputs is manageable. However, estimating PR across a large input space is challenging because enumeration becomes computationally infeasible. We propose OmniInput, a novel approach to evaluate and compare NNs by the PR of an input space. OmniInput is distinctive from previous works as its estimated PR reflects the estimation of the differences between human annotation and model prediction in the input space which is usually too huge to be enumerated. We empirically validate our method within an enumerable input space, and our experiments demonstrate that OmniInput can effectively estimate and compare precision and recall for (large) language models within a broad input space that is not enumerable.
BibTeX
@misc{liu2023evaluation,
  author = {Weitang Liu and Ying Wai Li and Tianle Wang and Yi‐Zhuang You and Jingbo Shang and Shang, Jingbo},
  title = {{Evaluation of human-model prediction difference on the Internet Scale of Data}},
  year = {2023},
  eprint = {2312.03291},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.03291},
}
Figure