Risk Consistent Multi-Class Learning from Label Proportions

Ryoma Kobayashi, Yusuke Mukuta, Tatsuya Harada

Abstract

This study addresses a multiclass learning from label proportions (MCLLP) setting in which training instances are provided in bags and only the proportion of each class within the bags is provided. Most existing MCLLP methods impose bag-wise constraints on the prediction of instances or assign them pseudo-labels; however, none of these methods have a theoretical consistency. To solve this problem, a risk-consistent method is proposed for instance classification using the empirical risk minimization framework, and its estimation error bound is derived. An approximation method is proposed for the proposed risk estimator, to apply it to large bags, by diverting the constraints on bags in existing research. The proposed method can be applied to any deep model or loss and is compatible with stochastic optimization. Experiments are conducted on benchmarks to verify the effectiveness of the proposed method.

Introduction

Learning from Label Proportions (LLP) is a weakly supervised learning paradigm where training data is organized into "bags," and only the label proportions for each bag are known, rather than individual instance labels. This setting is particularly relevant in privacy-sensitive domains like medical imaging or demographic analysis. While binary LLP has been extensively studied, Multiclass LLP (MCLLP) presents significant challenges, particularly regarding theoretical consistency and scalability with deep learning models.

Problem Statement

Existing MCLLP methods often suffer from two primary issues:

  1. High Variance and Instability: Heuristic methods that match bag-level predictions often produce high-variance gradients when paired with deep neural networks, leading to poor convergence.
  2. Theoretical Gap: Many current approaches lack statistical guarantees. There is no proof that these "per-bag" heuristics will converge to the optimal instance-level classifier as the amount of data increases.

Methodology

The authors propose two theoretically grounded frameworks to bridge the gap between bag-level observations and instance-level classification:

Risk-Consistent (RC) Method

The RC method is an instance-wise approach. It reformulates the standard supervised risk function into a form that can be computed using only bag proportions. By weighting potential label candidates based on the bag's distribution, it ensures that the empirical risk estimator is unbiased and consistent. Interestingly, the authors frame MCLLP as a variation of Partial Label Learning (PLL), where all labels in a bag are potential candidates for every instance, weighted by known proportions.

Classifier-Consistent (CC) Method

The CC method takes a bag-wise perspective. It treats label proportions as a multiset and aims to learn an instance classifier such that the expected bag-level proportions match the observed ones. This method is mathematically shown to be equivalent to training on fully supervised data under certain conditions.

Scalability via Approximation

A significant challenge in MCLLP is the computational cost of exact risk estimation, which scales exponentially with bag size $K$ and class count $C$ ($O(K^C)$). To address this, the authors introduce RC_Approx and CC_Approx. By utilizing a multinomial distribution approximation, the complexity is reduced to $O(KC)$, enabling the methods to scale to large bags and complex datasets.

Experimental Evaluation

The proposed methods were evaluated against several baselines, including DLLP, LLPFC, and Optimal Transport (OT), across benchmarks such as MNIST, Fashion-MNIST (F-MNIST), Kuzushiji-MNIST (K-MNIST), and CIFAR-10. The experiments covered various bag sizes ($K \in \{2, 4, \dots, 128\}$) and model architectures ranging from linear models to deep ResNets.

Table 1: Performance comparison (Accuracy ± SD) on MNIST, F-MNIST, and K-MNIST using linear models across different bag sizes K.

Key Results

The experimental results demonstrate several key advantages of the proposed consistent methods:

  • Superior Accuracy: As shown in Table 1, the RC and CC methods (and their approximations) consistently outperform or match state-of-the-art baselines across different datasets and bag sizes.
  • Robustness to Large Bags: Unlike previous methods that degrade rapidly as bag size increases, the proposed approximation methods maintain high performance. For instance, while linear models struggle at $K=128$, the authors note that RC_Approx with an MLP architecture maintains ~97.6% accuracy on MNIST even at that scale.
  • Deep Learning Compatibility: The risk-consistent formulation provides stable gradients, making it highly compatible with deep architectures where previous heuristics often failed to converge or produced biased results.

Conclusion

This work provides a solid theoretical foundation for Multiclass Learning from Label Proportions. By introducing risk-consistent and classifier-consistent estimators, the authors move beyond empirical heuristics toward statistically sound algorithms. The proposed approximation techniques further ensure that these methods are practical for large-scale deep learning applications. Future research will focus on developing approximation methods with provable theoretical guarantees.

Summary updated

Abstract

This study addresses a multiclass learning from label proportions (MCLLP) setting in which training instances are provided in bags and only the proportion of each class within the bags is provided. Most existing MCLLP methods impose bag-wise constraints on the prediction of instances or assign them pseudo-labels; however, none of these methods have a theoretical consistency. To solve this problem, a risk-consistent method is proposed for instance classification using the empirical risk minimization framework, and its estimation error bound is derived. An approximation method is proposed for the proposed risk estimator, to apply it to large bags, by diverting the constraints on bags in existing research. The proposed method can be applied to any deep model or loss and is compatible with stochastic optimization. Experiments are conducted on benchmarks to verify the effectiveness of the proposed method.
BibTeX
@misc{kobayashi2022risk,
  author = {Ryoma Kobayashi and Yusuke Mukuta and Tatsuya Harada},
  title = {{Risk Consistent Multi-Class Learning from Label Proportions}},
  year = {2022},
  eprint = {2203.12836},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.12836},
}

Loading related content...

Figure