Abstract
Merging the predictions of multiple experts is a frequent task. When ground-truth response values are available, this merging is often based on the estimated accuracies of the experts. In various applications, however, the only available information are the experts’ predictions on unlabeled test data, which do not allow to directly estimate their accuracies. Moreover, simple merging schemes such as majority voting in classification or the ensemble mean or median in regression, are clearly sub-optimal when some experts are more accurate than others. Focusing on regression tasks, in this work we propose U-PCR, a framework for unsupervised ensemble regression. Specifically, we develop spectral-based methods that under mild assumptions and in the absence of ground truth data, are able to estimate the mean squared error of the different experts and combine their predictions to a more accurate meta-learner. We provide theoretical support for U-PCR as well as empirical evidence for the validity of its underlying assumptions. On a variety of regression problems, we illustrate the improved accuracy of U-PCR over various unsupervised merging strategies. Finally, we also illustrate its applicability to unsupervised multi-class ensemble learning.
Introduction
Ensemble learning—combining multiple models to achieve superior performance—is a cornerstone of modern machine learning. Typically, this is performed in a supervised manner using a validation set to weight experts by their known accuracy. However, in many real-world scenarios, such as crowdsourcing or biological competitions, ground-truth labels are unavailable. In these unsupervised settings, simple strategies like the ensemble mean or median are common but sub-optimal, as they treat all experts equally regardless of their true reliability. This paper introduces U-PCR (Unsupervised Principal Component Regression), a method designed to identify and favor the most accurate experts using only their predictions on unlabeled data.
The U-PCR Framework
U-PCR approaches the problem through spectral analysis of the experts' covariance matrix ($C$). The core assumption is that $C$ can be decomposed into a low-rank matrix ($L$), representing the shared signal (the underlying truth), and a sparse matrix ($S$), representing individual and potentially correlated errors.
Unlike traditional models that assume independent errors, U-PCR allows for sparse correlations between experts (e.g., models sharing a common codebase). The algorithm follows four main steps:
- Extract the low-rank structure from the observed prediction covariance.
- Estimate the variance of the latent optimal predictor.
- Calculate the covariance between each expert and the hidden truth.
- Compute optimal weights to construct the final meta-learner.
This process requires only the experts' predictions, the expected mean of the target variable, and a rough bound on its variance.
Experimental Evaluation
The authors validated U-PCR across several domains:
- UCI Regression Datasets: 17 standard datasets covering diverse topics like energy output and flight delays.
- HPN-DREAM Challenge: A biological dataset for predicting protein concentrations in breast cancer cells.
- CIFAR-10C: A multiclass classification task converted to regression via probability vectors.
- Bounding Box Challenge: Coordinate prediction for object detection.
Results and Discussion
U-PCR consistently outperformed standard unsupervised baselines. On nearly all 17 regression tasks, it achieved a lower Mean Squared Error (MSE) than the simple mean or median. Notably, in 10 of these tasks, U-PCR matched or exceeded the performance of the single best expert in the pool—an expert whose identity is unknown in an unsupervised setting.
A key strength of U-PCR is its ability to rank experts by accurately estimating their individual MSEs without ground-truth labels. As shown in the figure above, the estimated MSE (y-axis) closely tracks the true MSE (x-axis) across various datasets, allowing the algorithm to effectively down-weight inaccurate contributors. In the CIFAR-10C task, this spectral approach reached 62% accuracy, significantly higher than the 58% achieved by majority voting.
Conclusion
U-PCR provides a robust framework for unsupervised ensemble learning by leveraging the spectral properties of expert predictions. While it requires prior knowledge of the response mean and variance bounds, its ability to handle correlated errors and accurately rank experts makes it a powerful tool for crowdsourced regression. Future work may explore semi-supervised extensions and more complex dependency structures between experts.
Summary updated