Common-Individual Semantic Fusion for Multi-View Multi-Label Learning

Gengyu Lyu, Weiqi Kang, Haobo Wang, Zheng Li, Zhen Yang, Songhe Feng

Abstract

In Multi-View Multi-Label Learning, each instance is described by several heterogeneous features and associated with multiple valid labels simultaneously. Existing methods mainly focus on leveraging feature-level view fusion to capture a common representation for multi-label classifier induction. In this paper, we take a new perspective and propose a new semantic-level fusion model named Common-Individual Semantic Fusion Multi-View Multi-Label Learning Method (CISF). Different from previous feature-level fusion model, our proposed method directly focuses on semantic-level view fusion and simultaneously take both the common semantic across different views and the individual semantic of each specific view into consideration. Specifically, we first assume each view involves some common semantic labels while owns a few exclusive semantic labels. Then, the common and exclusive semantic labels are separately forced to be consensus and diverse to excavate the consistences and complementarities among different views. Afterwards, we introduce the low-rank and sparse constraint to highlight the label co-occurrence relationship of common semantics and the view-specific expression of individual semantics. We provide theoretical guarantee for the strict convexity of our method by properly setting parameters. Extensive experiments on various data sets have verified the superiority of our method.

Introduction

In real-world applications, data is often complex and multifaceted. For example, a news webpage may consist of text, images, and video (Multi-View) and be associated with several topics like "Sports," "FIFA World Cup," and "Lionel Messi" (Multi-Label). The challenge in Multi-View Multi-Label Learning (MVML) lies in effectively combining these heterogeneous data sources.

Most existing methods rely on "feature-level fusion," which attempts to find a single unified representation for all views before inducing a classifier. However, this approach often fails to respect the unique characteristics of each view and can ignore rare labels. We propose Common-Individual Semantic Fusion (CISF), which shifts the focus to semantic-level fusion, explicitly separating shared information from view-specific details to create a more robust and accurate model.

The CISF Framework

The core philosophy of CISF is that every view contains two distinct types of information: Common Semantics, which are shared across all views, and Individual Semantics, which are unique to a specific view. By modeling these separately, the framework can capture both the consistency and the complementarity of multi-view data.

The CISF framework architecture illustrating the decomposition of multi-view data into common and individual semantic components.

Key Technical Components

  • Consensus via HSIC: To ensure that common semantics are truly shared, the model uses the Hilbert-Schmidt Independence Criterion (HSIC) to force different views to reach a consensus on core labels.
  • Diversity via Orthogonality: To ensure individual semantics are unique, the model minimizes the overlap between the individual label sets of different views.
  • Structural Constraints: We apply a low-rank constraint (nuclear norm) on common semantics to capture label correlations (e.g., if an instance is labeled "Soccer," it is likely also "Sports"). A sparse constraint is applied to individual semantics to highlight specific view-specific nuances.
  • Dynamic Label Correlation: Instead of a static map, the model learns a dynamic correlation matrix ($S$) during training to better adapt to the specific dataset.

Experimental Evaluation

We evaluated CISF on seven benchmark datasets, ranging from smaller tasks like Emotions and Scene to large-scale image datasets like Pascal and Mirflickr. The model was compared against five state-of-the-art methods (including FIMAN and BEMVL) across five standard metrics: Hamming Loss, Ranking Loss, One Error, Coverage, and Average Precision.

Key Results

CISF demonstrated dominant performance, achieving the best results in 97.14% of cases. Statistical tests, including the Friedman and Bonferroni-Dunn tests, confirmed that CISF consistently ranks 1st overall across all evaluation metrics.

Critical difference diagrams showing the statistical superiority and 1st-place ranking of CISF across multiple evaluation metrics.

Beyond accuracy, the model is computationally efficient, typically converging within 30 iterations. Theoretical analysis also guarantees the strict convexity of the optimization problem, ensuring the algorithm reliably finds the global optimum.

Conclusion and Future Work

CISF represents a significant step forward in MVML by prioritizing semantic-level fusion. While the model is highly effective, future research will address the computational complexity of Singular Value Decomposition (SVD) for very large label sets and adapt the framework to handle incomplete datasets where some views may be missing.

Summary updated

Abstract

In Multi-View Multi-Label Learning, each instance is described by several heterogeneous features and associated with multiple valid labels simultaneously. Existing methods mainly focus on leveraging feature-level view fusion to capture a common representation for multi-label classifier induction. In this paper, we take a new perspective and propose a new semantic-level fusion model named Common-Individual Semantic Fusion Multi-View Multi-Label Learning Method (CISF). Different from previous feature-level fusion model, our proposed method directly focuses on semantic-level view fusion and simultaneously take both the common semantic across different views and the individual semantic of each specific view into consideration. Specifically, we first assume each view involves some common semantic labels while owns a few exclusive semantic labels. Then, the common and exclusive semantic labels are separately forced to be consensus and diverse to excavate the consistences and complementarities among different views. Afterwards, we introduce the low-rank and sparse constraint to highlight the label co-occurrence relationship of common semantics and the view-specific expression of individual semantics. We provide theoretical guarantee for the strict convexity of our method by properly setting parameters. Extensive experiments on various data sets have verified the superiority of our method.
BibTeX
@inproceedings{lyu2024common,
  author = {Gengyu Lyu and Weiqi Kang and Haobo Wang and Zheng Li and Zhen Yang and Songhe Feng},
  title = {{Common-Individual Semantic Fusion for Multi-View Multi-Label Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=uMYmCgdL4q},
}
Figure