A Probabilistic Framework for Multi-Label Learning with Unseen Labels

Abhilash Gaure, Aishwarya Gupta, Vinay Kumar Verma, Piyush RaiUAI 2017 · OpenReview

Introduction

In modern information systems, objects such as images, videos, or articles are often associated with multiple tags simultaneously. Standard multi-label classification models typically require extensive training data for every possible label. However, in dynamic environments like content tagging or recommender systems, the vocabulary of labels is constantly expanding. Retraining models from scratch for every new tag is computationally prohibitive and data-intensive.

While zero-shot learning—the ability to predict classes not seen during training—has gained traction in multi-class settings, its application to multi-label problems remains challenging. Traditional approaches often overlook the structural correlations between labels. This paper introduces a probabilistic generative framework that leverages label co-occurrence statistics from external, unsupervised sources to predict unseen labels effectively.

Methodology

The proposed model, ML-LCS (Multi-Label Learning with Co-occurrence Statistics), utilizes a joint generative framework that integrates two distinct latent factor models to bridge the gap between seen and unseen categories.

The Generative Model

The framework consists of two primary components:

  1. Label Matrix Model: A logistic-Bernoulli model that maps input features to a binary matrix of seen labels from the training set.
  2. Co-occurrence Matrix Model: A negative binomial model that processes a count-valued matrix representing co-occurrences of both seen and unseen labels, mined from external corpora like Wikipedia.
Probabilistic graphical model of the ML-LCS framework.

The Zero-Shot Bridge

The model learns two types of embeddings. "Type-1" embeddings are derived for seen labels using both the label and co-occurrence matrices. "Type-2" embeddings are learned for all labels (seen and unseen) using only the co-occurrence matrix. A regression function is then trained to map Type-2 embeddings to Type-1 space. At test time, this mapping allows the model to synthesize Type-1 embeddings for unseen labels, enabling zero-shot prediction.

Efficient Inference

To handle the non-conjugacy of the logistic-Bernoulli and negative binomial distributions, the authors employ Pólya-gamma data augmentation. This technique transforms the likelihoods into a Gaussian form, allowing for an efficient Expectation-Maximization (EM) algorithm where updates are reduced to parallelizable ridge regressions.

Experimental Evaluation

The model was evaluated on three benchmark datasets: Bibtex (text), MediaMill (video), and Delicious (web bookmarks). The experiments simulated a zero-shot scenario by withholding a portion of the labels during the training phase.

Results

ML-LCS consistently outperformed baseline methods across all datasets. The inclusion of co-occurrence data not only enables zero-shot capabilities but also serves as a regularizer that improves performance on seen labels, particularly when training data is sparse.

Comparative performance of ML-LCS on benchmark datasets.

Key findings include:

  • Zero-Shot Superiority: On the Bibtex dataset, ML-LCS achieved a Precision@1 of 0.51, significantly higher than the COSTA baseline's 0.43.
  • Unseen Label Accuracy: When ranking only unseen labels, ML-LCS maintained a substantial lead, achieving a Precision@1 of 0.34 compared to 0.23 for COSTA.
  • Robustness: The choice of the negative binomial distribution proved superior to the Poisson distribution for handling the high variance (overdispersion) typical of count-based co-occurrence data.

Conclusion and Future Directions

This work demonstrates that unsupervised text statistics provide sufficient structural information to synthesize classifiers for new categories without manual attribute labeling. While the current EM approach is efficient, it is susceptible to local optima. Future research may explore non-linear mapping functions using Variational Autoencoders (VAEs) or integrating active learning to refine the model's understanding of the most informative unseen labels.

Summary updated

BibTeX
@inproceedings{gaure2017probabilistic,
  author = {Abhilash Gaure and Aishwarya Gupta and Vinay Kumar Verma and Piyush Rai},
  title = {{A Probabilistic Framework for Multi-Label Learning with Unseen Labels}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2017},
  url = {https://openreview.net/forum?id=fyzI6IzV00},
}
Figure