Semi-Supervised Multi-Modal Learning with Balanced Spectral Decomposition

Peng Hu, Hongyuan Zhu, Xi Peng, Jie Lin

Introduction: Bridging the Heterogeneous Gap

In our increasingly digital world, we are inundated with multimedia data spanning various formats like images, text, and audio. Cross-modal retrieval—the ability to search for a text description using an image or vice versa—is a fundamental task. However, it remains a significant challenge due to the heterogeneous gap, where different modalities reside in distinct mathematical spaces.

At AAAI-20, Peng Hu and colleagues introduced the Semi-supervised Multimodal Learning Network (SMLN). This framework addresses the gap by rethinking how latent spaces are optimized where disparate modalities meet and interact.

The SMLN Framework

The core objective of SMLN is to project different data modalities into a shared latent space that captures both labeled discriminative information and unlabeled geometric structures.

Architecture of the SMLN framework

As shown in the architecture above, SMLN consists of several key components:

  • Modality-Specific Networks: Individual neural networks (e.g., CNNs for images, Doc2Vec for text) project each data type into a shared $d$-dimensional space.
  • Unified Similarity Matrix ($W$): This matrix integrates cross-modal correlations, label-based discrimination, and intra-modal graph information to preserve the underlying data structure.
  • End-to-End Learning: Unlike traditional spectral methods that require global optimization over the entire dataset, SMLN integrates its objective directly into a neural network loss, enabling scalable batch-by-batch training.

Innovation: Balanced Spectral Decomposition

A common issue in latent space optimization is the "Dominant Eigenvalue" problem. Standard methods often maximize a ratio trace criterion, which tends to focus on the largest eigenvalues. In a deep learning context, this causes the network to ignore "minor" eigenvalues that often contain fine-grained discriminative information.

To solve this, the authors proposed Balanced Spectral Decomposition. Instead of direct maximization, they minimize a novel objective function: $\sum_{i=1}^d e^{-\alpha \lambda_i}$.

Visualization of the exponential loss function

As illustrated in the figure above, the exponential decay function acts as a balancer. For small eigenvalues ($\lambda_i$), the gradient is large, forcing the network to pay attention to these subtle features. As an eigenvalue becomes dominant (larger), its gradient shrinks, preventing it from monopolizing the learning process. This ensures a more robust, multi-dimensional representation where even minor features contribute to discrimination.

Experimental Evaluation

The researchers evaluated SMLN on three benchmark datasets: Wikipedia, NUS-WIDE, and XMediaNet, using varying proportions of labeled data (5%, 10%, and 30%).

Key Results

SMLN consistently outperformed 13 state-of-the-art methods, including deep supervised models like DCCA and ACMR.

  • Low-Label Performance: On the challenging XMediaNet dataset with only 5% labeled data, SMLN achieved a mean Average Precision (mAP) of 0.612. This significantly outperformed traditional methods like MCCA (0.355) and other deep semi-supervised approaches.
  • Balanced Retrieval: The model showed consistent performance across both Image-to-Text (Img2Txt) and Text-to-Image (Txt2Img) tasks, confirming the creation of a truly common latent space.
  • Ablation Success: Experiments confirmed that both the geometric structure preservation and the balanced loss function are critical to the model's success.

Conclusion

SMLN represents a major advancement in semi-supervised cross-modal retrieval. By rethinking spectral decomposition through a balanced loss function, it forces neural networks to learn more nuanced representations that go beyond dominant features. Furthermore, its batch-wise optimization makes graph-based semi-supervised learning viable for large-scale web data. Future work may extend this "balancing" approach to more diverse modalities such as 3D point clouds or sensor data.

Reference: Hu, P., Zhu, H., Peng, X., & Lin, J. (2020). Semi-Supervised Multi-Modal Learning with Balanced Spectral Decomposition. Proceedings of the AAAI Conference on Artificial Intelligence.

Summary updated

Abstract

Cross-modal retrieval aims to retrieve the relevant samples across different modalities, of which the key problem is how to model the correlations among different modalities while narrowing the large heterogeneous gap. In this paper, we propose a Semi-supervised Multimodal Learning Network method (SMLN) which correlates different modalities by capturing the intrinsic structure and discriminative correlation of the multimedia data. To be specific, the labeled and unlabeled data are used to construct a similarity matrix which integrates the cross-modal correlation, discrimination, and intra-modal graph information existing in the multimedia data. What is more important is that we propose a novel optimization approach to optimize our loss within a neural network which involves a spectral decomposition problem derived from a ratio trace criterion. Our optimization enjoys two advantages given below. On the one hand, the proposed approach is not limited to our loss, which could be applied to any case that is a neural network with the ratio trace criterion. On the other hand, the proposed optimization is different from existing ones which alternatively maximize the minor eigenvalues, thus overemphasizing the minor eigenvalues and ignore the dominant ones. In contrast, our method will exactly balance all eigenvalues, thus being more competitive to existing methods. Thanks to our loss and optimization strategy, our method could well preserve the discriminative and instinct information into the common space and embrace the scalability in handling large-scale multimedia data. To verify the effectiveness of the proposed method, extensive experiments are carried out on three widely-used multimodal datasets comparing with 13 state-of-the-art approaches.
BibTeX
@inproceedings{hu2020semi,
  author = {Peng Hu and Hongyuan Zhu and Xi Peng and Jie Lin},
  title = {{Semi-Supervised Multi-Modal Learning with Balanced Spectral Decomposition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i01.5339},
}
Figure