Abstract
Weakly supervised learning aims to empower machine learning when perfect supervision is unavailable. Among various types of weak supervision, one of the most challenging cases is learning from multiple unlabeled (U) datasets with only limited knowledge of class priors, known as $U^m$ learning. This paper introduces $U^m$-AUC, a framework for building an AUC optimization model from multiple unlabeled datasets. By converting the $U^m$ data into a multi-label AUC optimization problem, the proposed approach achieves efficient training and robust performance.
Introduction
In many practical applications, obtaining labeled data is costly, leading to the prevalence of weak supervision. $U^m$ learning involves $m$ unlabeled datasets, each with distinct but often unknown proportions of positive and negative instances. While standard classification accuracy has been the focus of previous $U^m$ research, it often fails in imbalanced scenarios. Area Under the ROC Curve (AUC) is a superior metric for such cases as it measures the model's ability to rank a random positive instance higher than a random negative one. However, optimizing AUC with unlabeled data is non-trivial due to the lack of explicit labels for pairwise comparisons.
The $U^m$-AUC Framework
The proposed $U^m$-AUC method addresses the mathematical challenges of AUC optimization in the $U^m$ setting through several key innovations.
Consistency with Relative Priors
A significant hurdle in $U^m$ learning is the requirement for exact class priors for each dataset. $U^m$-AUC demonstrates that exact values are unnecessary; instead, only the relative order of these priors is required to achieve AUC consistency. If one can rank the datasets by their "positivity," an optimal ranking model can be recovered.
Multi-label Transformation and Efficiency
A naive approach to AUC optimization involves comparing all possible pairs of instances, resulting in a quadratic $O(n^2)$ time complexity. $U^m$-AUC cleverly transforms the problem into a multi-label AUC optimization task with $m-1$ surrogate labels. This transformation allows for solving multiple subproblems simultaneously.
To further enhance scalability, the objective is reformulated as a stochastic saddle point (min-max) problem. This enables the use of point-wise stochastic gradient updates, reducing training complexity to linear $O(n)$, making it suitable for large-scale datasets.
Empirical Evaluation
The authors evaluated $U^m$-AUC using synthesized $U^m$ scenarios from three benchmarks: Kuzushiji-MNIST (K-MNIST), CIFAR-10, and CIFAR-100. They simulated various prior distributions—Uniform ($\\mathcal{D}_u$), Biased ($\\mathcal{D}_b$), Concentrated ($\\mathcal{D}_c$), and Biased-Concentrated ($\\mathcal{D}_{bc}$)—to test robustness across different dataset compositions.
Key Results
As shown in the table below, $U^m$-AUC consistently outperforms state-of-the-art $U^m$ classification methods such as LLP-VAT and $U^m$-SSC across all datasets and prior distributions.
For instance, on CIFAR-10 with $m=10$, $U^m$-AUC achieved an AUC of 0.905, significantly higher than the best baseline's 0.860. The framework also maintained high stability when the number of unlabeled sets was increased to $m=50$.
Conclusion
$U^m$-AUC provides a theoretically sound and computationally efficient solution for AUC optimization in the challenging $U^m$ learning setting. By leveraging the relative order of class priors and a multi-label transformation, it achieves superior performance and scalability. Future work may explore extending this framework to multi-class AUC optimization and investigating alternative surrogate loss functions.
Summary updated