AUC Optimization from Multiple Unlabeled Datasets

Yu Liu, Zheng Xie, Ming Li

Abstract

Weakly supervised learning aims to empower machine learning when perfect supervision is unavailable. Among various types of weak supervision, one of the most challenging cases is learning from multiple unlabeled (U) datasets with only limited knowledge of class priors, known as $U^m$ learning. This paper introduces $U^m$-AUC, a framework for building an AUC optimization model from multiple unlabeled datasets. By converting the $U^m$ data into a multi-label AUC optimization problem, the proposed approach achieves efficient training and robust performance.

Introduction

In many practical applications, obtaining labeled data is costly, leading to the prevalence of weak supervision. $U^m$ learning involves $m$ unlabeled datasets, each with distinct but often unknown proportions of positive and negative instances. While standard classification accuracy has been the focus of previous $U^m$ research, it often fails in imbalanced scenarios. Area Under the ROC Curve (AUC) is a superior metric for such cases as it measures the model's ability to rank a random positive instance higher than a random negative one. However, optimizing AUC with unlabeled data is non-trivial due to the lack of explicit labels for pairwise comparisons.

The $U^m$-AUC Framework

The proposed $U^m$-AUC method addresses the mathematical challenges of AUC optimization in the $U^m$ setting through several key innovations.

Consistency with Relative Priors

A significant hurdle in $U^m$ learning is the requirement for exact class priors for each dataset. $U^m$-AUC demonstrates that exact values are unnecessary; instead, only the relative order of these priors is required to achieve AUC consistency. If one can rank the datasets by their "positivity," an optimal ranking model can be recovered.

Multi-label Transformation and Efficiency

A naive approach to AUC optimization involves comparing all possible pairs of instances, resulting in a quadratic $O(n^2)$ time complexity. $U^m$-AUC cleverly transforms the problem into a multi-label AUC optimization task with $m-1$ surrogate labels. This transformation allows for solving multiple subproblems simultaneously.

Comparison between the naive solution and the proposed Um-AUC framework.

To further enhance scalability, the objective is reformulated as a stochastic saddle point (min-max) problem. This enables the use of point-wise stochastic gradient updates, reducing training complexity to linear $O(n)$, making it suitable for large-scale datasets.

Empirical Evaluation

The authors evaluated $U^m$-AUC using synthesized $U^m$ scenarios from three benchmarks: Kuzushiji-MNIST (K-MNIST), CIFAR-10, and CIFAR-100. They simulated various prior distributions—Uniform ($\\mathcal{D}_u$), Biased ($\\mathcal{D}_b$), Concentrated ($\\mathcal{D}_c$), and Biased-Concentrated ($\\mathcal{D}_{bc}$)—to test robustness across different dataset compositions.

Key Results

As shown in the table below, $U^m$-AUC consistently outperforms state-of-the-art $U^m$ classification methods such as LLP-VAT and $U^m$-SSC across all datasets and prior distributions.

Experimental results comparing Um-AUC with baseline methods across different datasets and prior distributions.

For instance, on CIFAR-10 with $m=10$, $U^m$-AUC achieved an AUC of 0.905, significantly higher than the best baseline's 0.860. The framework also maintained high stability when the number of unlabeled sets was increased to $m=50$.

Conclusion

$U^m$-AUC provides a theoretically sound and computationally efficient solution for AUC optimization in the challenging $U^m$ learning setting. By leveraging the relative order of class priors and a multi-label transformation, it achieves superior performance and scalability. Future work may explore extending this framework to multi-class AUC optimization and investigating alternative surrogate loss functions.

Summary updated

Abstract

Weakly supervised learning aims to empower machine learning when the perfect supervision is unavailable, which has drawn great attention from researchers. Among various types of weak supervision, one of the most challenging cases is to learn from multiple unlabeled (U) datasets with only a little knowledge of the class priors, or U$^m$ learning for short. In this paper, we study the problem of building an AUC (area under ROC curve) optimization model from multiple unlabeled datasets, which maximizes the pairwise ranking ability of the classifier. We propose U$^m$-AUC, an AUC optimization approach that converts the U$^m$ data into a multi-label AUC optimization problem, and can be trained efficiently. We show that the proposed U$^m$-AUC is effective theoretically and empirically.
BibTeX
@inproceedings{liu2024auc,
  author = {Yu Liu and Zheng Xie and Ming Li},
  title = {{AUC Optimization from Multiple Unlabeled Datasets}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i14.29538},
}
Figure