OmniInput: A Model-centric Evaluation Framework through Output Distribution

Weitang Liu, Ying Wai Li, Tianle Wang, Yi-Zhuang You, Jingbo ShangSubmission to ICLR 2024 · Rejected

TL;DR

OmniInput is a model-centric evaluation framework that assesses neural classifiers across their entire input space rather than relying on curated, fixed test sets. By coupling Wang–Landau physical sampling algorithms with selective human annotation, it constructs comprehensive precision-recall curves that expose severe overconfidence and hidden failure modes missed by standard benchmarks.

What Problem Does This Paper Address?

Traditional machine learning evaluations are data-centric: models are graded against static, human-annotated test sets. However, strong in-distribution (and even curated out-of-distribution) test accuracy often hides brittle failure modes. Deep networks are notorious for overconfident predictions, frequently assigning near-100% confidence to uninformative noise, adversarial perturbations, or nonsensical inputs.

Furthermore, out-of-distribution (OOD) benchmarks are inconsistent: a model's ranking can fluctuate wildly depending on which arbitrary OOD dataset is selected. The authors tackle this blind spot by proposing a way to evaluate models over their entire input space ($\Omega = \{0, \dots, N\}^D$, which for even a simple $28 \times 28$ 8-bit image contains $256^{784}$ possible inputs—far exceeding the number of atoms in the universe).

Methods

OmniInput shifts evaluation from data-centric to model-centric using four primary steps:

  1. Output Distribution Sampling: The framework treats the model's output distribution $\rho(z)$—the frequency of all possible inputs mapping to logit $z$—similarly to the physical "density of states." Using the Gradient Wang–Landau (GWL) sampler, it uniformly navigates the model’s logit space, capturing representative input samples across discrete output bins.
  2. Selective Annotation: Evaluators inspect a small set of representative inputs sampled per bin (e.g., 40–50 samples) and assign a score $r(z) \in [0, 1]$ indicating whether the input matches human judgment for the target class.
  3. Distribution-Weighted Metrics: Precision and unnormalized recall are estimated for any logit threshold $\lambda$ by weighting bin scores against the global output distribution: $$\text{precision}_\lambda = \frac{\sum_{z \ge \lambda} r(z)\rho(z)}{\sum_{z \ge \lambda} \rho(z)}, \quad \text{recall}_\lambda \propto \sum_{z \ge \lambda} r(z)\rho(z)$$
  4. Full-Space Precision-Recall Curves: Sweeping $\lambda$ generates a precision-recall curve representing performance over the complete input space.
[GWL Sampler] ---> [Output Distribution ρ(z) & Representative Inputs] 
              ---> [Human Annotation r(z)] 
              ---> [Weighted Full-Space PR Curve]

Datasets

The authors evaluated models across both synthetic full spaces and traditional datasets:

  • MNIST (0 vs. 1): Classifiers and energy-based generative models evaluated over the full $256^{28 \times 28}$ image space.
  • OOD Benchmarks (for comparison): Fashion-MNIST, Kuzushiji-MNIST, EMNIST (byclass split), and Q-MNIST.
  • CIFAR-10 (Airplane vs. Automobile): ResNet binary classifier evaluated over the discrete $32 \times 32 \times 3$ image space.
  • SST2: DistilBERT sentiment classifier evaluated over fixed-length token sequences (lengths 10 and 66).

Key Insights and Ideas

  • Standard Benchmarks Can Hide Catastrophic Flaws: Models achieving >97% or even >99% test accuracy on MNIST can have near-zero precision over the full input space. For example, a vanilla CNN learned to classify almost any dark-background noise pattern as the digit "1."
  • Data Augmentation Beats Pure Generative Priors in Recall: While energy-based generative models produced realistic digits at high logits, they exhibited low recall. Conversely, training with uniform noise data augmentation (RES-AUG-MNIST-0/1) allowed the model to maintain high precision and substantially broader recall.
  • Generative Evaluation Metrics (like FID) Fail in OOD Spaces: Automated metrics like Fréchet Inception Distance (FID) gave misleadingly favorable scores to random noise outputs because feature extractors (e.g., Inception) are themselves uncertified outside their training distribution.
  • Convergence of Optimal Classifiers and Generators: The authors speculate that a theoretically perfect open-world classifier and a perfect generative model converge to the same functional mapping.

Key Results

  • Inconsistent Traditional Rankings: On pre-defined test sets, model rankings reversed depending on the dataset. A generative model (RES-GEN-MNIST-1) ranked 3rd on in-distribution test data (99.99% AUPR) but 1st across all OOD sets (e.g., 100.00% on Fashion-MNIST, 99.87% on EMNIST).
  • OmniInput PR Performance: OmniInput revealed that RES-AUG-MNIST-0/1 was the superior model overall. In contrast, standard CNNs (CNN-MNIST-0/1 and CNN-AUG-MNIST-0/1) exhibited precision near 0.0 across high logits due to massive overconfidence on non-digit noise.
  • CIFAR-10 & DistilBERT Failures:
  • A ResNet on CIFAR-10 with 93.34% standard accuracy yielded virtually 0 precision across its 299 sampled bins.
  • DistilBERT on SST2 (91% accuracy) scored near-zero precision on full sequence spaces, picking up isolated positive keywords while remaining oblivious to syntax or coherence.
  • Low Human Annotation Burden: Annotation stabilized rapidly; only 40 to 50 samples per bin were necessary for the precision-recall curve to converge.

Limitations and Future Work

  • Sampler Scalability: The framework relies heavily on advanced MCMC algorithms (like GWL). Sampling the output distribution of massive vision models or modern Large Language Models (LLMs) remains computationally demanding.
  • Task Scope: Experiments are currently limited to binary classification and short text sequence lengths.
  • Future Directions: Developing faster, massively parallel samplers and extending OmniInput to multi-class and multi-dimensional output spaces.

Summary updated

Abstract

We propose a novel model-centric evaluation framework, OmniInput, to evaluate the quality of an AI/ML model's predictions on all possible inputs (including human-unrecognizable ones), which is crucial for AI safety and reliability. Unlike traditional data-centric evaluation based on pre-defined test sets, the test set in OmniInput is self-constructed by the model itself and the model quality is evaluated by investigating its output distribution. We employ an efficient sampler to obtain representative inputs and the output distribution of the trained model, which, after selective annotation, can be used to estimate the model's precision and recall at different output values and a comprehensive precision-recall curve. Our experiments demonstrate that OmniInput enables a more fine-grained comparison between models, especially when their performance is almost the same on pre-defined test sets, leading to new findings and insights for how to train more robust, generalizable models.
BibTeX
@misc{liu2023omniinput,
  author = {Weitang Liu and Ying Wai Li and Tianle Wang and Yi-Zhuang You and Jingbo Shang},
  title = {{OmniInput: A Model-centric Evaluation Framework through Output Distribution}},
  year = {2023},
  eprint = {2312.03291},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.03291},
}
Figure