Evaluating multiple models using labeled and unlabeled data

Divya Shanmugam, Shuvom Sadhuka, Manish Raghavan, John Guttag, Bonnie Berger, Emma Pierson

TL;DR

Evaluating machine learning models usually requires a mountain of expensive labeled data. This paper introduces Semi-Supervised Model Evaluation (SSME), a framework that combines a tiny sliver of labeled data with abundant unlabeled data to accurately estimate classifier performance. By modeling the joint distribution of scores across multiple classifiers simultaneously, SSME reduces performance estimation error by 5.1× compared to using labeled data alone.

What Problem Does This Paper Address?

Model evaluation is a massive bottleneck in modern machine learning. With the rise of model hubs, practitioners have access to a myriad of pre-trained classifiers but often lack the labeled data required to verify their accuracy, calibration, or area under the ROC curve (AUC).

While unlabeled data is usually plentiful, using it for evaluation is incredibly challenging because the ground truth labels are missing. Prior approaches typically evaluate only a single model at a time, throw away continuous probability scores in favor of hard discrete labels, or rely purely on small labeled datasets, leading to high variance and inaccurate performance estimates. This paper tackles the challenge of how to reliably evaluate multiple models by unlocking the hidden signals inside abundant unlabeled datasets.

Methods

The authors propose Semi-Supervised Model Evaluation (SSME), which treats evaluation as a joint density estimation problem. Instead of looking at one model in isolation, SSME aggregates continuous prediction scores from multiple classifiers at the same time.

The method operates in two main steps:

  1. Estimating $P(y, s)$: It fits a semi-supervised mixture model over the concatenated classifier scores ($s$) and ground truth labels ($y$), where each mixture component corresponds to a true class. To avoid "boundary bias" (since probabilities are bounded between 0 and 1), the scores are mapped to unbounded real numbers using an additive log-ratio (ALR) transform. The mixture model is optimized using the Expectation-Maximization (EM) algorithm, leveraging a non-parametric Kernel Density Estimator (KDE) to capture complex score distributions.
  2. Computing Metrics: Once the joint distribution is learned, SSME infers the conditional distribution $P(y|s)$ for the unlabeled data. It samples pseudo-labels for the unlabeled points, combines them with the true labels from the small labeled dataset, and computes any standard metric (e.g., Accuracy, AUC, Expected Calibration Error).
Figure 1: Using SSME with two binary classifiers. SSME applies to settings in which classifier scores are available for a set of examples, some labeled (left). SSME first fits a mixture model to estimate the joint density of scores and labels (where s1 is the score assigned by the first classifier, and s2 the second, middle panel). SSME next uses the mixture model to compute metrics with respect to the expected label for each example (right). SSME can be applied to any number of classifiers and classes K, and supports any metric that compares classifier scores to ground truth labels.

Datasets

The authors evaluated SSME across five diverse, real-world datasets spanning healthcare, text classification, and chemistry:

  • MIMIC-IV: Electronic health records used for predicting three clinical outcomes (hospitalization, critical outcome, emergency department revisits).
  • CivilComments: Social media comments flagged for toxicity detection.
  • OGB-SARS-CoV: A molecular graph dataset for biochemical property prediction (SARS-CoV inhibition).
  • MultiNLI: A natural language inference dataset evaluated using off-the-shelf language models.
  • AG News: A news classification task evaluated using open- and closed-source Large Language Models (LLMs).

Key Insights and Ideas

  • The Power of the Ensemble's Signal: A key "aha" moment is that multiple classifiers provide complementary information about the latent true label. Even without knowing the label of an unlabeled point, looking at where it falls in the multi-dimensional score space of all classifiers makes it much easier to infer its true class.
  • Continuous Scores Matter: Discretizing probabilities into hard predictions throws away valuable confidence information. SSME models continuous probabilities directly, which proves crucial for estimating calibration metrics like Expected Calibration Error (ECE).
  • Theoretical Guarantees: The authors mathematically prove that performance estimation error decreases as you add more unlabeled data and more classifiers, provided the new classifiers bring additional separation between the classes.

Key Results

  • Massive Error Reductions: With access to just 20 labeled examples and 1,000 unlabeled examples, SSME reduced metric estimation error by 5.1× relative to using labeled data alone, and 2.4× relative to the next-best baseline method.
  • High Absolute Accuracy: On average, SSME estimated classifier accuracy to within 1.5 percentage points of the true ground truth.
  • Superb Calibration Estimation: SSME showed massive gains in estimating Expected Calibration Error (ECE)—reducing error by 7.2× over labeled data alone—a metric that is traditionally highly volatile with small sample sizes.
  • Subgroup Fairness & LLMs: In case studies, SSME effectively estimated performance across demographic subgroups (age, sex, race) to audit algorithmic fairness and successfully evaluated downstream classifiers derived from LLMs.
Figure 2: Metric estimation error on binary tasks (nℓ= 20, nu = 1000). Each point plots the rescaled mean absolute error (RMAE) across runs, where 1.0 (dashed line) represents the RMAE when using labeled data alone and lower is better. SSME (gray) achieves lower estimation error than the baselines, averaging across metrics and 50 runs, reported for five binary tasks (y-axis). Results reporting mean absolute error (unscaled) and 95% CIs are in Tables S3-S6.

Limitations and Future Work

  • The Curse of Dimensionality: The current implementation relies on Kernel Density Estimation (KDE), which scales poorly as the number of classifiers or target classes grows large. The authors suggest exploring more scalable density estimators (like normalizing flows) in the future.
  • Assumption of Identical Distributions: SSME assumes that the labeled and unlabeled data come from the exact same distribution. In many real-world scenarios, unlabeled data suffers from distribution shift, which would violate this assumption and degrade performance. Extending SSME to handle covariate and label shifts is a prominent direction for future research.

Summary updated

Abstract

It remains difficult to evaluate machine learning classifiers in the absence of a large, labeled dataset. While labeled data can be prohibitively expensive or impossible to obtain, unlabeled data is plentiful. Here, we introduce Semi-Supervised Model Evaluation (SSME), a method that uses both labeled and unlabeled data to evaluate machine learning classifiers. SSME is the first evaluation method to take advantage of the fact that: (i) there are frequently multiple classifiers for the same task, (ii) continuous classifier scores are often available for all classes, and (iii) unlabeled data is often far more plentiful than labeled data. The key idea is to use a semi-supervised mixture model to estimate the joint distribution of ground truth labels and classifier predictions. We can then use this model to estimate any metric that is a function of classifier scores and ground truth labels (e.g., accuracy or expected calibration error). We present experiments in four domains where obtaining large labeled datasets is often impractical: (1) healthcare, (2) content moderation, (3) molecular property prediction, and (4) image annotation. Our results demonstrate that SSME estimates performance more accurately than do competing methods, reducing error by 5.1x relative to using labeled data alone and 2.4x relative to the next best competing method. SSME also improves accuracy when evaluating performance across subsets of the test distribution (e.g., specific demographic subgroups) and when evaluating the performance of language models.
BibTeX
@inproceedings{shanmugam2025evaluating,
  author = {Divya Shanmugam and Shuvom Sadhuka and Manish Raghavan and John Guttag and Bonnie Berger and Emma Pierson},
  title = {{Evaluating multiple models using labeled and unlabeled data}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=BpwIPxSSdb},
}
Figure