Adaptively Unified Semi-supervised Learning for Cross-Modal Retrieval

Liang Zhang, Bingpeng Ma, Jianfeng He, Guorong Li, Qingming Huang, Qi Tian

Abstract

Cross-modal retrieval aims to bridge the "heterogeneity gap" between different data modalities, such as images and text. While supervised methods rely on expensive manual annotations, semi-supervised approaches often struggle with computational efficiency and the quality of unlabeled data. This paper introduces Adaptively Unified Semi-supervised Learning (AUSL), a novel framework that leverages both labeled and unlabeled data through a unified linear regression model. AUSL incorporates an $\epsilon$-dragging technique to enhance class discriminability and an adaptive weighting mechanism to prioritize high-quality unlabeled samples. Experimental results on benchmark datasets demonstrate that AUSL significantly outperforms state-of-the-art methods, even when using a fraction of the available labels.

Introduction

The core challenge in cross-modal retrieval is the heterogeneity gap: images and text reside in distinct feature spaces, making direct comparison impossible. Traditional supervised methods map these modalities into a common subspace using class labels. However, obtaining large-scale labeled datasets is labor-intensive. Semi-supervised learning (SSL) offers a solution by utilizing unlabeled data, but many existing SSL methods rely on complex graph-based models that are computationally heavy and difficult to scale to new "out-of-sample" data.

The AUSL Framework

The proposed Adaptively Unified Semi-supervised Learning (AUSL) framework addresses these challenges by integrating three innovative components into a joint optimization problem.

Figure 1: Illustration of mapping data from the original heterogeneous space (a) to a unified common subspace (b) using the AUSL framework.
  1. Unified Linear Regression with $\epsilon$-Dragging: Unlike standard regression that only pulls samples toward their correct labels, AUSL employs "dragging." This technique forces features toward relevant labels while simultaneously pushing them away from irrelevant ones, which is particularly beneficial for multi-label scenarios.
  2. Adaptive Weighted Regression: Not all unlabeled data is equally informative. AUSL distinguishes between "specific points" (samples with clear semantic membership) and "non-specific points" (ambiguous samples). By adaptively weighting these samples, the model ensures that only high-quality semantic information from the unlabeled pool influences the learning process.
  3. Feature Selection: The model utilizes $\ell_{2,1}$-norm regularization to identify the most discriminative features, reducing noise and preventing overfitting.

Experimental Evaluation

AUSL was evaluated on three standard benchmarks: Wiki, Pascal VOC, and NUS-WIDE. The framework was tested under challenging conditions, using only 80% of training labels (with 20% unlabeled), while competing methods often used 100% labeled data.

Despite the reduced label count, AUSL achieved superior Mean Average Precision (MAP) across all datasets. On the large-scale NUS-WIDE dataset, AUSL reached a MAP of 0.4909, outperforming the next best method, LGCFL (0.4440). t-SNE visualizations confirmed that AUSL effectively clusters labeled data and correctly aligns "specific" unlabeled points within the common subspace.

Conclusion

AUSL provides a robust and efficient solution for semi-supervised cross-modal retrieval. By combining discriminative "dragging" with selective utilization of unlabeled data, it achieves state-of-the-art performance with lower annotation costs. Future work may explore extending these principles to non-linear deep learning architectures to further enhance retrieval accuracy in complex, high-dimensional spaces.

Summary updated

Abstract

Motivated by the fact that both relevancy of class labels and unlabeled data can help to strengthen multi-modal correlation, this paper proposes a novel method for cross-modal retrieval. To make each sample moving to the direction of its relevant label while far away from that of its irrelevant ones, a novel dragging technique is fused into a unified linear regression model. By this way, not only the relation between embedded features and relevant class labels but also the relation between embedded features and irrelevant class labels can be exploited. Moreover, considering that some unlabeled data contain specific semantic information, a weighted regression model is designed to adaptively enlarge their contribution while weaken that of the unlabeled data with non-specific semantic information. Hence, unlabeled data can supply semantic information to enhance discriminant ability of classifier. Finally, we integrate the constraints into a joint minimization formulation and develop an efficient optimization algorithm to learn a discriminative common subspace for different modalities. Experimental results on Wiki, Pascal and NUS-WIDE datasets show that the proposed method outperforms the state-of-the-art methods even when we set 20% samples without class labels.
BibTeX
@inproceedings{zhang2017adaptively,
  author = {Liang Zhang and Bingpeng Ma and Jianfeng He and Guorong Li and Qingming Huang and Qi Tian},
  title = {{Adaptively Unified Semi-supervised Learning for Cross-Modal Retrieval}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/476},
}
Figure