Privileged label enhancement with multi-label learning

Wenfang Zhu, Xiuyi Jia, Weiwei LiIJCAI 2020 · OpenReview

Abstract

Label distribution learning has attracted more and more attention in view of its more generalized ability to express the label ambiguity. However, it is much more expensive to obtain the label distribution information of the data rather than the logical labels. Thus, label enhancement is proposed to recover the label distributions from the logical labels. In this paper, we propose a novel label enhancement method by using privileged information. We first apply a multi-label learning model to implicitly capture the complex structural information between instances and generate the privileged information. Second, we adopt LUPI (learning with privileged information) paradigm to utilize the privileged information and employ RSVM+ as the prediction model. Finally, comparison experiments on 12 datasets demonstrate that our proposal can better fit the ground-truth label distributions.

Introduction

In traditional Multi-Label Learning (MLL), an instance is typically assigned a set of logical labels: a binary "1" for presence or "0" for absence. However, this rigid classification fails to capture label ambiguity. For example, a facial expression might be 70% "happy" and 30% "surprised." Label Distribution Learning (LDL) addresses this by assigning a real number to every label, representing its degree of description.

Because label distributions are difficult and expensive to collect manually, researchers use Label Enhancement (LE) algorithms to "recover" these hidden distributions from existing binary datasets. While most current LE methods rely on simple geometric assumptions like the manifold hypothesis, this paper introduces PLEML, a framework designed to mine complex structural relationships between features and labels to create more accurate distributions.

Methodology

The PLEML framework operates in two distinct stages, utilizing a "teacher-student" dynamic to refine label distributions.

1. Privileged Information Learning

The first stage involves training a multi-label learning model with a low-rank constraint (using the trace norm). This constraint forces the model to capture correlations between different labels (e.g., "smiling" and "happy" often co-occur). The soft predictions generated by this model are treated as Privileged Information (PI)—additional "teacher" knowledge that describes the underlying data structure more deeply than binary labels.

2. Learning Using Privileged Information (LUPI)

Inspired by Vladimir Vapnik’s LUPI paradigm, the authors employ the RSVM+ (Regression Support Vector Machine Plus) model. In this stage, the PI from Stage 1 guides the training of the final prediction model. Crucially, this "teacher" information is only required during training; the final model remains efficient and can be applied to new data where privileged information is unavailable.

The PLEML framework: Stage 1 generates privileged information via an MLL model, and Stage 2 utilizes this information to train the RSVM+ prediction model.

Optimization is achieved using the Alternating Direction Method of Multipliers (ADMM) for the first stage and a penalty function method for the second.

Experimental Setup

The researchers evaluated PLEML on 12 real-world datasets:

  • Facial Expressions: SJAFFE and SBU_3DFE, where human emotions are naturally ambiguous.
  • Biological Data: 10 Yeast datasets (e.g., Yeast-cold, Yeast-heat) involving gene functional analysis where different functions have varying degrees of relevance.

Key Results and Discussion

PLEML was compared against five state-of-the-art LE algorithms (including GLLE and ML) across six metrics, such as KL divergence, Cosine similarity, and Chebyshev distance.

Experimental results across 12 datasets and 6 evaluation metrics. PLEML consistently outperforms state-of-the-art label enhancement algorithms.
  • Superior Accuracy: PLEML achieved the best performance on the vast majority of datasets. For instance, on the Yeast-dtt dataset, PLEML's KL divergence was significantly lower than its closest competitor, GLLE.
  • Implicit Correlation: By using a low-rank structure, the model successfully learned that labels are not independent, preventing the enhancement process from creating noisy or unrealistic distributions.
  • Downstream Success: When recovered distributions were used to train an LDL classifier (LDL-SCL), models trained on PLEML-enhanced data performed nearly as well as those trained on "ground-truth" distributions.

Conclusion and Future Work

While PLEML shows significant promise, the authors noted sensitivity to small datasets. On the SJAFFE dataset (213 instances), the multi-label model occasionally struggled to capture accurate mapping relationships, leading to noisier privileged information. Additionally, the two-stage optimization process is more computationally intensive than simpler graph-based methods.

Future research will explore alternative ways to generate privileged information, potentially moving beyond linear models to deep learning architectures for the "teacher" phase.

Summary updated

Abstract

Label distribution learning has attracted more and more attention in view of its more generalized ability to express the label ambiguity. However, it is much more expensive to obtain the label distribution information of the data rather than the logical labels. Thus, label enhancement is proposed to recover the label distributions from the logical labels. In this paper, we propose a novel label enhancement method by using privileged information. We first apply a multi-label learning model to implicitly capture the complex structural information between instances and generate the privileged information. Second, we adopt LUPI (learning with privileged information) paradigm to utilize the privileged information and employ RSVM+ as the prediction model. Finally, comparison experiments on 12 datasets demonstrate that our proposal can better fit the ground-truth label distributions.
BibTeX
@inproceedings{zhu2020privileged,
  author = {Wenfang Zhu and Xiuyi Jia and Weiwei Li},
  title = {{Privileged label enhancement with multi-label learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=KYWVV_DnqCk},
}
Figure