Multi-Label Open Set Recognition

Yibo Wang, Jun-Yi Hang, Min-Ling ZhangNeurIPS 2024 · OpenReview

The Challenge of Open-World Multi-Label Learning

Traditional multi-label learning (MLL) assumes a "closed set" environment where all possible labels are known during training. For instance, an image might be tagged with "sky", "car", and "tree." However, real-world data is dynamic. At test time, that same image might contain those known labels alongside a completely novel concept, like "drone" or "trestle bridge."

Example of Multi-Label Open Set Recognition

Open Set Recognition (OSR) tackles unseen classes, but existing methods are designed for single-label, multi-class problems (e.g., an image is either a car or a drone). They fail in multi-label scenarios because unknown labels almost always co-occur with known ones. It is incredibly difficult to separate an instance containing only known labels from one containing known labels plus an unknown label. To bridge this gap, this paper formalizes the Multi-Label Open Set Recognition (MLOSR) framework, aiming to correctly classify known labels while simultaneously flagging the presence of unknown ones.

The SLAN Approach

To solve the MLOSR problem, the authors propose SLAN (Sub-Labeling informAtion reconstructioN). The core idea is to differentiate between holistic label supervision and "sub-labeling" information to detect anomalies.

SLAN operates through three main mechanisms:

  1. Structural Information Discovery: SLAN captures the underlying geometry (manifold structure) of the feature space by constructing a weighted directed graph of the training instances using global sparse reconstruction.
  2. Sub-Labeling Information Enrichment: Instead of relying solely on binary labels, SLAN transforms them into numerical vectors and examines a "sub-label space" (the label space minus one specific label). By reconstructing instances in this sub-label space and comparing them to the holistic label space, the model establishes a baseline for known label combinations. Significant deviations from this baseline signal the presence of an unknown label.
  3. Joint Optimization: Rather than training an anomaly detector and a multi-label classifier separately, SLAN optimizes them simultaneously using an Alternating Direction Method of Multipliers (ADMM) and gradient descent. This joint training allows the classifier to adjust its boundaries to account for open-space risk, making it far more robust.

Key Insights

  • The Co-occurrence Challenge: The biggest hurdle in MLOSR is that unknown labels hide behind known ones. Because an instance with an unknown label might still share a dense feature region with known labels, standard anomaly detectors tend to fail or falsely reject valid known labels.
  • Sub-labeling as an Anomaly Proxy: By systematically removing one label at a time and evaluating how well the feature manifold can reconstruct the instance, SLAN creates a clever proxy for anomaly detection. It essentially asks: "Given the feature structure, does this test instance's label distribution make sense based on our training data?"

Experimental Results

The authors evaluated SLAN on nine diverse benchmark datasets (e.g., enron, corel5k, bibtex), simulating an open-set environment by treating 50% of the labels as "unknown" during testing.

  • Multi-Label Classification: Even with open-set instances introduced, SLAN achieved superior or comparable performance to state-of-the-art closed-set MLL baselines in 91.7% of the evaluated cases across metrics like Ranking loss, Average precision, and Macro-averaging AUC.
  • Unknown Label Recognition: When tasked with detecting unknown labels, SLAN significantly outperformed traditional anomaly detection approaches.
F-measure comparison for unknown label recognition

As shown in the results, SLAN outperformed baselines like Isolation Forest and One-Class SVM in 83.3% of cases (measured via F-measure). This proves that single-label anomaly detectors are ill-equipped for multi-label co-occurrences, whereas SLAN's joint optimization excels.

Limitations and Future Work

  • Scalability: SLAN enriches #labels + 1 sub-labeling information matrices. This scales poorly and would struggle with "extreme multi-label" datasets containing tens of thousands of classes.
  • Feature Representation: Currently relying on standard feature mappings, future work could explore deep MLOSR approaches to learn more discriminative representations directly from raw data (like images or text).
  • Metrics: There is an ongoing need to develop evaluation metrics specifically tailored for the unique nuances of the MLOSR problem.

Summary updated

Abstract

In multi-label learning, each training instance is associated with multiple labels simultaneously. Traditional multi-label learning studies primarily focus on closed set scenario, i.e. the class label set of test data is identical to those used in training phase. Nevertheless, in numerous real-world scenarios, the environment is open and dynamic where unknown labels may emerge gradually during testing. In this paper, the problem of multi-label open set recognition (MLOSR) is investigated, which poses significant challenges in classifying and recognizing instances with unknown labels in multi-label setting. To enable open set multi-label prediction, a novel approach named SLAN is proposed by leveraging sub-labeling information enriched by structural information in the feature space. Accordingly, unknown labels are recognized by differentiating the sub-labeling information from holistic supervision. Experimental results on various datasets validate the effectiveness of the proposed approach in dealing with the MLOSR problem.
BibTeX
@inproceedings{wang2024multi,
  author = {Yibo Wang and Jun-Yi Hang and Min-Ling Zhang},
  title = {{Multi-Label Open Set Recognition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=k1VrxRS6WZ},
}
Figure