TL;DR
Evaluating machine learning models usually requires a mountain of expensive labeled data. This paper introduces Semi-Supervised Model Evaluation (SSME), a framework that combines a tiny sliver of labeled data with abundant unlabeled data to accurately estimate classifier performance. By modeling the joint distribution of scores across multiple classifiers simultaneously, SSME reduces performance estimation error by 5.1× compared to using labeled data alone.
What Problem Does This Paper Address?
Model evaluation is a massive bottleneck in modern machine learning. With the rise of model hubs, practitioners have access to a myriad of pre-trained classifiers but often lack the labeled data required to verify their accuracy, calibration, or area under the ROC curve (AUC).
While unlabeled data is usually plentiful, using it for evaluation is incredibly challenging because the ground truth labels are missing. Prior approaches typically evaluate only a single model at a time, throw away continuous probability scores in favor of hard discrete labels, or rely purely on small labeled datasets, leading to high variance and inaccurate performance estimates. This paper tackles the challenge of how to reliably evaluate multiple models by unlocking the hidden signals inside abundant unlabeled datasets.
Methods
The authors propose Semi-Supervised Model Evaluation (SSME), which treats evaluation as a joint density estimation problem. Instead of looking at one model in isolation, SSME aggregates continuous prediction scores from multiple classifiers at the same time.
The method operates in two main steps:
- Estimating $P(y, s)$: It fits a semi-supervised mixture model over the concatenated classifier scores ($s$) and ground truth labels ($y$), where each mixture component corresponds to a true class. To avoid "boundary bias" (since probabilities are bounded between 0 and 1), the scores are mapped to unbounded real numbers using an additive log-ratio (ALR) transform. The mixture model is optimized using the Expectation-Maximization (EM) algorithm, leveraging a non-parametric Kernel Density Estimator (KDE) to capture complex score distributions.
- Computing Metrics: Once the joint distribution is learned, SSME infers the conditional distribution $P(y|s)$ for the unlabeled data. It samples pseudo-labels for the unlabeled points, combines them with the true labels from the small labeled dataset, and computes any standard metric (e.g., Accuracy, AUC, Expected Calibration Error).
Datasets
The authors evaluated SSME across five diverse, real-world datasets spanning healthcare, text classification, and chemistry:
- MIMIC-IV: Electronic health records used for predicting three clinical outcomes (hospitalization, critical outcome, emergency department revisits).
- CivilComments: Social media comments flagged for toxicity detection.
- OGB-SARS-CoV: A molecular graph dataset for biochemical property prediction (SARS-CoV inhibition).
- MultiNLI: A natural language inference dataset evaluated using off-the-shelf language models.
- AG News: A news classification task evaluated using open- and closed-source Large Language Models (LLMs).
Key Insights and Ideas
- The Power of the Ensemble's Signal: A key "aha" moment is that multiple classifiers provide complementary information about the latent true label. Even without knowing the label of an unlabeled point, looking at where it falls in the multi-dimensional score space of all classifiers makes it much easier to infer its true class.
- Continuous Scores Matter: Discretizing probabilities into hard predictions throws away valuable confidence information. SSME models continuous probabilities directly, which proves crucial for estimating calibration metrics like Expected Calibration Error (ECE).
- Theoretical Guarantees: The authors mathematically prove that performance estimation error decreases as you add more unlabeled data and more classifiers, provided the new classifiers bring additional separation between the classes.
Key Results
- Massive Error Reductions: With access to just 20 labeled examples and 1,000 unlabeled examples, SSME reduced metric estimation error by 5.1× relative to using labeled data alone, and 2.4× relative to the next-best baseline method.
- High Absolute Accuracy: On average, SSME estimated classifier accuracy to within 1.5 percentage points of the true ground truth.
- Superb Calibration Estimation: SSME showed massive gains in estimating Expected Calibration Error (ECE)—reducing error by 7.2× over labeled data alone—a metric that is traditionally highly volatile with small sample sizes.
- Subgroup Fairness & LLMs: In case studies, SSME effectively estimated performance across demographic subgroups (age, sex, race) to audit algorithmic fairness and successfully evaluated downstream classifiers derived from LLMs.
Limitations and Future Work
- The Curse of Dimensionality: The current implementation relies on Kernel Density Estimation (KDE), which scales poorly as the number of classifiers or target classes grows large. The authors suggest exploring more scalable density estimators (like normalizing flows) in the future.
- Assumption of Identical Distributions: SSME assumes that the labeled and unlabeled data come from the exact same distribution. In many real-world scenarios, unlabeled data suffers from distribution shift, which would violate this assumption and degrade performance. Extending SSME to handle covariate and label shifts is a prominent direction for future research.
Summary updated