Abstract
Cross-modal retrieval aims to bridge the "heterogeneity gap" between different data modalities, such as images and text. While supervised methods rely on expensive manual annotations, semi-supervised approaches often struggle with computational efficiency and the quality of unlabeled data. This paper introduces Adaptively Unified Semi-supervised Learning (AUSL), a novel framework that leverages both labeled and unlabeled data through a unified linear regression model. AUSL incorporates an $\epsilon$-dragging technique to enhance class discriminability and an adaptive weighting mechanism to prioritize high-quality unlabeled samples. Experimental results on benchmark datasets demonstrate that AUSL significantly outperforms state-of-the-art methods, even when using a fraction of the available labels.
Introduction
The core challenge in cross-modal retrieval is the heterogeneity gap: images and text reside in distinct feature spaces, making direct comparison impossible. Traditional supervised methods map these modalities into a common subspace using class labels. However, obtaining large-scale labeled datasets is labor-intensive. Semi-supervised learning (SSL) offers a solution by utilizing unlabeled data, but many existing SSL methods rely on complex graph-based models that are computationally heavy and difficult to scale to new "out-of-sample" data.
The AUSL Framework
The proposed Adaptively Unified Semi-supervised Learning (AUSL) framework addresses these challenges by integrating three innovative components into a joint optimization problem.
- Unified Linear Regression with $\epsilon$-Dragging: Unlike standard regression that only pulls samples toward their correct labels, AUSL employs "dragging." This technique forces features toward relevant labels while simultaneously pushing them away from irrelevant ones, which is particularly beneficial for multi-label scenarios.
- Adaptive Weighted Regression: Not all unlabeled data is equally informative. AUSL distinguishes between "specific points" (samples with clear semantic membership) and "non-specific points" (ambiguous samples). By adaptively weighting these samples, the model ensures that only high-quality semantic information from the unlabeled pool influences the learning process.
- Feature Selection: The model utilizes $\ell_{2,1}$-norm regularization to identify the most discriminative features, reducing noise and preventing overfitting.
Experimental Evaluation
AUSL was evaluated on three standard benchmarks: Wiki, Pascal VOC, and NUS-WIDE. The framework was tested under challenging conditions, using only 80% of training labels (with 20% unlabeled), while competing methods often used 100% labeled data.
Despite the reduced label count, AUSL achieved superior Mean Average Precision (MAP) across all datasets. On the large-scale NUS-WIDE dataset, AUSL reached a MAP of 0.4909, outperforming the next best method, LGCFL (0.4440). t-SNE visualizations confirmed that AUSL effectively clusters labeled data and correctly aligns "specific" unlabeled points within the common subspace.
Conclusion
AUSL provides a robust and efficient solution for semi-supervised cross-modal retrieval. By combining discriminative "dragging" with selective utilization of unlabeled data, it achieves state-of-the-art performance with lower annotation costs. Future work may explore extending these principles to non-linear deep learning architectures to further enhance retrieval accuracy in complex, high-dimensional spaces.
Summary updated