Abstract
Multi-label feature selection often suffers from the presence of missing labels in training data. Traditional methods either treat missing labels as negative or attempt unreliable imputation, both of which introduce noise. This paper introduces Generative Multi-label Feature Selection (GMFS), a probabilistic framework that explicitly models label observability. By utilizing a spike-and-slab prior for variable selection and Polya-gamma augmentation for efficient inference, GMFS effectively identifies relevant features even under high label sparsity. Experimental results across diverse datasets demonstrate that GMFS significantly outperforms state-of-the-art methods in both accuracy and robustness.
Introduction
In multi-label learning, instances are associated with multiple tags simultaneously. However, obtaining complete label sets is often impractical due to the high cost of manual annotation or the "laziness" of human labelers. This leads to a significant portion of labels being unobserved. Standard feature selection techniques typically handle missing labels by either assuming they are negative (the "pessimist" approach) or predicting their values (the "guesser" approach). Both strategies are prone to introducing noise, particularly for rare but critical labels.
The GMFS Framework
To address these challenges, the authors propose GMFS (Generative Multi-label Feature Selection). Unlike discriminative models, GMFS treats feature selection as a probabilistic generative process.
The model incorporates three primary components:
- Label Observability Modeling: Instead of imputing missing values, GMFS estimates the probability that a label was observed. It leverages a relaxed "Smoothness Assumption," where similar instances are expected to have similar labeling probabilities.
- Spike-and-Slab Prior: Recognized as a robust Bayesian variable selection tool, this prior uses a mixture of a "spike" (narrow distribution at zero) to prune irrelevant features and a "slab" (wider distribution) to maintain important ones. This provides superior noise shrinkage compared to standard $L_1$ regularization.
- Efficient Inference via Data Augmentation: The complex generative math is handled using Polya-gamma augmentation, enabling a fast Expectation-Maximization (EM) algorithm with closed-form updates.
Experimental Results
The model was evaluated on seven multi-label datasets (e.g., Emotions, Yeast, Bibtex) with simulated label missingness at 20% and 40% levels.
Robustness to Missing Labels
GMFS demonstrates remarkable resilience to label sparsity. While competing methods like FSNM and MLMLFS show significant performance degradation as the percentage of missing labels increases, GMFS maintains high Mean Average Precision (MAP) and low Hamming Loss.
Key Insights
- Tracing vs. Recovering: By tracing unobserved labels rather than hallucinating their values through imputation, GMFS avoids learning from incorrect data.
- Selective Shrinkage: The spike-and-slab prior acts as a surgical tool, preserving features discriminative for rare labels that are often suppressed by standard Lasso-based methods.
Conclusion
GMFS provides a robust solution for multi-label feature selection in the presence of missing data. By modeling the generative process of labels and their observability, it achieves superior accuracy and sparsity. Future work will focus on scaling the algorithm for massive datasets via online EM and incorporating complex label hierarchies.
Summary updated