To Avoid the Pitfall of Missing Labels in Feature Selection: A Generative Model Gives the Answer

Yuanyuan Xu, Jun Wang, Jinmao WeiAAAI 2020 · DOI

Abstract

Multi-label feature selection often suffers from the presence of missing labels in training data. Traditional methods either treat missing labels as negative or attempt unreliable imputation, both of which introduce noise. This paper introduces Generative Multi-label Feature Selection (GMFS), a probabilistic framework that explicitly models label observability. By utilizing a spike-and-slab prior for variable selection and Polya-gamma augmentation for efficient inference, GMFS effectively identifies relevant features even under high label sparsity. Experimental results across diverse datasets demonstrate that GMFS significantly outperforms state-of-the-art methods in both accuracy and robustness.

Introduction

In multi-label learning, instances are associated with multiple tags simultaneously. However, obtaining complete label sets is often impractical due to the high cost of manual annotation or the "laziness" of human labelers. This leads to a significant portion of labels being unobserved. Standard feature selection techniques typically handle missing labels by either assuming they are negative (the "pessimist" approach) or predicting their values (the "guesser" approach). Both strategies are prone to introducing noise, particularly for rare but critical labels.

The GMFS Framework

To address these challenges, the authors propose GMFS (Generative Multi-label Feature Selection). Unlike discriminative models, GMFS treats feature selection as a probabilistic generative process.

Graphical representation of the GMFS model, illustrating the interaction between feature selection, label matrix, and label observability components.

The model incorporates three primary components:

  1. Label Observability Modeling: Instead of imputing missing values, GMFS estimates the probability that a label was observed. It leverages a relaxed "Smoothness Assumption," where similar instances are expected to have similar labeling probabilities.
  2. Spike-and-Slab Prior: Recognized as a robust Bayesian variable selection tool, this prior uses a mixture of a "spike" (narrow distribution at zero) to prune irrelevant features and a "slab" (wider distribution) to maintain important ones. This provides superior noise shrinkage compared to standard $L_1$ regularization.
  3. Efficient Inference via Data Augmentation: The complex generative math is handled using Polya-gamma augmentation, enabling a fast Expectation-Maximization (EM) algorithm with closed-form updates.

Experimental Results

The model was evaluated on seven multi-label datasets (e.g., Emotions, Yeast, Bibtex) with simulated label missingness at 20% and 40% levels.

Robustness to Missing Labels

GMFS demonstrates remarkable resilience to label sparsity. While competing methods like FSNM and MLMLFS show significant performance degradation as the percentage of missing labels increases, GMFS maintains high Mean Average Precision (MAP) and low Hamming Loss.

Performance comparison (Hamming Loss and MAP) of GMFS against baseline methods across varying percentages of missing labels.

Key Insights

  • Tracing vs. Recovering: By tracing unobserved labels rather than hallucinating their values through imputation, GMFS avoids learning from incorrect data.
  • Selective Shrinkage: The spike-and-slab prior acts as a surgical tool, preserving features discriminative for rare labels that are often suppressed by standard Lasso-based methods.

Conclusion

GMFS provides a robust solution for multi-label feature selection in the presence of missing data. By modeling the generative process of labels and their observability, it achieves superior accuracy and sparsity. Future work will focus on scaling the algorithm for massive datasets via online EM and incorporating complex label hierarchies.

Summary updated

BibTeX
@inproceedings{xu2020avoid,
  author = {Yuanyuan Xu and Jun Wang and Jinmao Wei},
  title = {{To Avoid the Pitfall of Missing Labels in Feature Selection: A Generative Model Gives the Answer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.6127},
}
Figure