Common-Individual Semantic Fusion for Multi-View Multi-Label Learning

Gengyu Lyu, Weiqi Kang, Haobo Wang, Zheng Li, Zhen Yang

Abstract

Multi-view multi-label (MVML) learning aims to predict multiple labels for objects described by diverse data sources, such as text, images, and video. Traditional feature-level fusion methods often struggle to balance shared information with view-specific nuances, potentially drowning out rare labels. This paper introduces Common-Individual Semantic Fusion (CISF), a novel framework that performs fusion at the semantic (label) level. By explicitly decomposing representations into common (consensus) and individual (complementary) semantics, CISF effectively captures both shared patterns and unique view-specific details. Experimental results across seven benchmark datasets demonstrate that CISF consistently outperforms state-of-the-art methods across multiple evaluation metrics, including Ranking Loss and Average Precision.

Introduction

In the real world, objects are frequently characterized by multiple "views." For instance, a news webpage might contain textual descriptions, accompanying images, and video clips. In Multi-View Multi-Label (MVML) learning, the objective is to leverage these diverse views to predict a set of relevant labels, such as "Sports," "FIFA World Cup," or "Lionel Messi."

Most existing MVML methods rely on feature-level fusion, attempting to compress different data types into a single shared latent space before label prediction. However, this approach has two primary drawbacks: it can obscure rare labels that only appear in specific views, and it fails to explicitly distinguish between information shared across all views (consensus) and information unique to a single view (complementarity).

Methodology: The CISF Framework

The authors propose the Common-Individual Semantic Fusion (CISF) method to address these limitations. Instead of fusing raw features, CISF maps each view directly into the semantic (label) space. The resulting representation is decomposed into three distinct components:

  1. Common Semantics ($C$): This captures "consensus" information. For example, if both an image and text suggest the label "Football," it is treated as a common semantic. The model uses the Hilbert-Schmidt Independence Criterion (HSIC) to enforce consistency across views and applies a low-rank constraint (nuclear norm) to reflect real-world label co-occurrence patterns.
  2. Individual Semantics ($D$): This captures "exclusive" information unique to a specific view, such as a stadium shown in a video but not mentioned in the text. These representations are forced to be diverse (orthogonal) and sparse to ensure they do not replicate the common semantics.
  3. Global Label Correlations ($S$): A dynamic correlation matrix is learned to capture global relationships between labels, ensuring the integrity of the final predictions.
The CISF architecture decomposes multi-view data into common and individual semantic components to improve label prediction.

The optimization of these components is handled via an Alternating Lagrange Multiplier (ALM) strategy, which iteratively updates the mapping matrices for each view.

Experimental Setup

The CISF model was evaluated on seven widely-used MVML datasets, ranging from small-scale sets like Emotions and Scene (6 labels) to larger image-based datasets like Corel5k, Espgame, and Iaprtc12 (hundreds of labels and 4-6 views). The testing ground also included Pascal and Mirflickr, which feature up to 457 labels. These datasets provide a robust evaluation of the algorithm's versatility across varying instance counts and feature dimensions.

Results and Discussion

CISF was compared against five state-of-the-art methods: LSPC, FIMAN, ICM2L, BEMVL, and NAIM3L. The results indicate a clear dominance by the proposed method:

  • Performance: CISF ranked 1st across all five evaluation metrics (Hamming Loss, Ranking Loss, One Error, Coverage, and Average Precision) on nearly every dataset.
  • Statistical Significance: Friedman and Bonferroni-Dunn tests confirmed that CISF is statistically superior to existing feature-fusion approaches.
  • Ablation Study: Removing either the "Common" or "Individual" components resulted in a performance drop, validating the necessity of both consensus and complementarity for accurate prediction.
Ranking loss comparison across benchmarks shows CISF consistently achieving the top rank compared to other state-of-the-art models.

Conclusion

By shifting fusion from the feature level to the semantic level, CISF provides a more interpretable and effective framework for multi-modal data. While the current model relies on linear mapping and involves computationally intensive Singular Value Decomposition (SVD), it sets a new benchmark for MVML tasks. Future work may explore integrating deep neural networks to capture non-linear relationships and optimizing the framework for extreme multi-label tasks involving millions of labels.

Summary updated

BibTeX
@inproceedings{lyu2024common,
  author = {Gengyu Lyu and Weiqi Kang and Haobo Wang and Zheng Li and Zhen Yang},
  title = {{Common-Individual Semantic Fusion for Multi-View Multi-Label Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=uMYmCgdL4q},
}
Figure