Introduction
In modern information systems, objects such as images, videos, or articles are often associated with multiple tags simultaneously. Standard multi-label classification models typically require extensive training data for every possible label. However, in dynamic environments like content tagging or recommender systems, the vocabulary of labels is constantly expanding. Retraining models from scratch for every new tag is computationally prohibitive and data-intensive.
While zero-shot learning—the ability to predict classes not seen during training—has gained traction in multi-class settings, its application to multi-label problems remains challenging. Traditional approaches often overlook the structural correlations between labels. This paper introduces a probabilistic generative framework that leverages label co-occurrence statistics from external, unsupervised sources to predict unseen labels effectively.
Methodology
The proposed model, ML-LCS (Multi-Label Learning with Co-occurrence Statistics), utilizes a joint generative framework that integrates two distinct latent factor models to bridge the gap between seen and unseen categories.
The Generative Model
The framework consists of two primary components:
- Label Matrix Model: A logistic-Bernoulli model that maps input features to a binary matrix of seen labels from the training set.
- Co-occurrence Matrix Model: A negative binomial model that processes a count-valued matrix representing co-occurrences of both seen and unseen labels, mined from external corpora like Wikipedia.
The Zero-Shot Bridge
The model learns two types of embeddings. "Type-1" embeddings are derived for seen labels using both the label and co-occurrence matrices. "Type-2" embeddings are learned for all labels (seen and unseen) using only the co-occurrence matrix. A regression function is then trained to map Type-2 embeddings to Type-1 space. At test time, this mapping allows the model to synthesize Type-1 embeddings for unseen labels, enabling zero-shot prediction.
Efficient Inference
To handle the non-conjugacy of the logistic-Bernoulli and negative binomial distributions, the authors employ Pólya-gamma data augmentation. This technique transforms the likelihoods into a Gaussian form, allowing for an efficient Expectation-Maximization (EM) algorithm where updates are reduced to parallelizable ridge regressions.
Experimental Evaluation
The model was evaluated on three benchmark datasets: Bibtex (text), MediaMill (video), and Delicious (web bookmarks). The experiments simulated a zero-shot scenario by withholding a portion of the labels during the training phase.
Results
ML-LCS consistently outperformed baseline methods across all datasets. The inclusion of co-occurrence data not only enables zero-shot capabilities but also serves as a regularizer that improves performance on seen labels, particularly when training data is sparse.
Key findings include:
- Zero-Shot Superiority: On the Bibtex dataset, ML-LCS achieved a Precision@1 of 0.51, significantly higher than the COSTA baseline's 0.43.
- Unseen Label Accuracy: When ranking only unseen labels, ML-LCS maintained a substantial lead, achieving a Precision@1 of 0.34 compared to 0.23 for COSTA.
- Robustness: The choice of the negative binomial distribution proved superior to the Poisson distribution for handling the high variance (overdispersion) typical of count-based co-occurrence data.
Conclusion and Future Directions
This work demonstrates that unsupervised text statistics provide sufficient structural information to synthesize classifiers for new categories without manual attribute labeling. While the current EM approach is efficient, it is susceptible to local optima. Future research may explore non-linear mapping functions using Variational Autoencoders (VAEs) or integrating active learning to refine the model's understanding of the most informative unseen labels.
Summary updated