Batch Selection for Multi-Label Classification Guided by Uncertainty and Dynamic Label Correlations

Ao Zhou, Bin Liu, Jin Wang, Grigorios TsoumakasAAAI 2025 · OpenReview

TL;DR

This paper introduces a novel batch selection strategy for multi-label deep learning that goes beyond simple uncertainty measures. By considering both label-wise prediction fluctuations and dynamic label correlations, their method significantly improves model performance and accelerates convergence across various datasets and architectures.

What Problem Does This Paper Address?

Training deep neural networks (DNNs) heavily relies on how data is presented in mini-batches. While effective batch selection strategies exist for single-label classification (e.g., prioritizing uncertain or difficult samples), the multi-label domain has lagged. Existing multi-label batch selection methods either ignore uncertainty or don't fully leverage it. Specifically, traditional uncertainty measures can be too simplistic, failing to capture the nuances of prediction volatility over time or the intricate relationships between labels. This paper aims to bridge this gap by developing a more sophisticated uncertainty-based batch selection method tailored for multi-label classification.

Methods

The core of this work lies in a two-pronged approach to uncertainty estimation for multi-label data:

  1. Enhanced Label Uncertainty: Instead of just looking at the current prediction's confidence, the authors propose a measure that combines:
  2. Current Prediction Confidence: Using entropy-based measures on the current epoch's predictions.
  3. Fine-grained Historical Fluctuations: Analyzing the absolute differences between consecutive predictions within a sliding window. This captures how volatile a label's prediction has been recently.
  4. Dynamic Uncertainty-Based Label Correlations: Recognizing that labels are not independent, the paper introduces a method to capture how label uncertainties co-vary. They compute mutual information between the binned uncertainty distributions of different labels. This "label correlation matrix" is then used to re-weight the individual label uncertainties, creating a sample uncertainty score that emphasizes instances where uncertainty is synergistically high across multiple, correlated labels.

Finally, these sample uncertainty scores are used to guide the selection of mini-batches. A "selection pressure" parameter is introduced, which decays over training. This helps to initially focus on highly uncertain samples and then gradually broaden the selection to more diverse samples, preventing overfitting and promoting smoother convergence.

Datasets

The paper evaluates their method on a variety of multi-label datasets, including:

  • Card: A dataset for image classification.
  • yeast: A biology-related dataset.
  • Corel5k: An image dataset.
  • rcv1subset1, rcv1subset2, rcv1subset3: Text classification datasets.
  • yahoo-Arts, yahoo-Business: Text classification datasets.
  • bibtex: A dataset related to academic publications.
  • tmc2007: A text classification dataset.
  • enron: A text classification dataset.
  • cal500: A dataset with a large number of labels.
  • LLOG-F: A text classification dataset.

The paper does not introduce new benchmarks but uses existing, well-established multi-label datasets to demonstrate the generalizability of their approach.

Key Insights and Ideas

  • Beyond Simple Uncertainty: The paper convincingly argues that traditional uncertainty measures are insufficient for multi-label tasks. Capturing the dynamics of prediction changes over time is crucial.
  • Label Correlations Matter: Ignoring label dependencies leads to suboptimal batch selection. The proposed method leverages these correlations to better understand sample difficulty.
  • Synergistic Uncertainty: The idea that uncertainty across multiple, related labels is more informative than the sum of individual uncertainties is a key insight.
  • Dynamic Selection Pressure: The decaying selection pressure is a clever mechanism to balance exploration (focusing on uncertain samples) and exploitation (ensuring diverse data exposure) throughout training.
  • Universality: The method's effectiveness across different base models (MPVAE, CLIF, DELA) and diverse datasets highlights its robustness and broad applicability.

Key Results

  • Performance Improvement: The proposed "Ours" method consistently achieves the best Macro-AUC scores across most datasets and base classifiers compared to Random, Balance, Active, Recent, and Hard batch selection methods.
  • Faster Convergence: Empirical results (shown in convergence curves) indicate that their method leads to faster convergence of the model's performance metrics.
  • Robustness: The method demonstrates consistent superiority across various deep multi-label learning models and datasets from different domains, suggesting it's not overfitted to a specific architecture or data type.
  • Outperforms State-of-the-Art: The paper claims their method outperforms five other batch selection methods, particularly on challenging datasets with large scales, high label dimensions, or significant imbalance.

Limitations and Future Work

  • Temperature Scaling: The authors suggest that temperature scaling could be explored to further improve the entropy-based uncertainty assessment.
  • Bayesian Averaging: They also propose considering theoretical uncertainty methods based on Bayesian averaging, such as Monte Carlo dropout, for future work.
  • Computational Complexity: While not explicitly detailed as a limitation, the introduction of dynamic label correlation calculations adds computational overhead compared to simpler methods. The paper mentions detailed empirical analyses in appendices, which might shed more light on this.

Summary updated

Abstract

The accuracy of deep neural networks is significantly influenced by the effectiveness of mini-batch construction during training. In single-label scenarios, such as binary and multi-class classification tasks, it has been demonstrated that batch selection algorithms preferring samples with higher uncertainty achieve better performance than difficulty-based methods. Although there are two batch selection methods tailored for multi-label data, none of them leverage important uncertainty information. Adapting the concept of uncertainty to multi-label data is not a trivial task, since there are two issues that should be tackled. First, traditional variance or entropy-based uncertainty measures ignore fluctuations of predictions within sliding windows and the importance of the current model state. Second, existing multi-label methods do not explicitly exploit the label correlations, particularly the uncertainty-based label correlations that evolve during the training process. In this paper, we propose an uncertainty-based multi-label batch selection algorithm. It assesses uncertainty for each label by considering differences between successive predictions and the confidence of current outputs, and further leverages dynamic uncertainty-based label correlations to emphasize instances whose uncertainty is synergistically expressed across multiple labels. Empirical studies demonstrate the effectiveness of our method in improving the performance and accelerating the convergence of various multi-label deep learning models.
BibTeX
@inproceedings{zhou2025batch,
  author = {Ao Zhou and Bin Liu and Jin Wang and Grigorios Tsoumakas},
  title = {{Batch Selection for Multi-Label Classification Guided by Uncertainty and Dynamic Label Correlations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=YOVjw808WH},
}
Figure