Group-Invariant Cross-Modal Subspace Learning

Jian Liang, Ran He, Zhenan Sun, Tieniu Tan

Abstract

Cross-modal retrieval typically relies on supervised learning with expensive semantic labels. This paper presents Simultaneous Pairwise and Groupwise Correspondences Maximization (SPGCM), an unsupervised framework that discovers latent semantic structures without manual annotation. By assuming paired data belong to shared latent groups, SPGCM learns a common subspace that captures broader neighborhood relationships. Experimental results on benchmark datasets demonstrate that SPGCM significantly outperforms traditional unsupervised methods and achieves performance competitive with supervised techniques.

Introduction

Cross-modal retrieval—using a text query to find images or vice versa—is a cornerstone of modern search. Most high-performing algorithms are supervised, meaning they require human-annotated semantic labels to learn how different data types relate. Existing unsupervised methods, like Canonical Correlation Analysis (CCA), typically rely on pairwise correspondences: they try to force a specific image and its specific caption to be close in a shared mathematical space. However, this approach is brittle as it ignores the broader "neighborhood" of the data—for example, the fact that all "cat" images and "cat" descriptions should cluster together even without a label. The challenge is to discover these latent semantic structures automatically.

Methodology: The SPGCM Framework

The authors propose a novel framework called SPGCM. Its core innovation is the introduction of groupwise correspondences to supplement traditional pairwise matching.

Conceptual comparison between CCA and SPGCM. CCA focuses on pairwise alignment, while SPGCM introduces latent groups (K) to capture broader semantic structures.

The Model

Instead of just matching pairs, SPGCM assumes there are $K$ latent groups in the data. It seeks a common subspace where paired heterogeneous data are not only close to each other but also belong to the same latent cluster. They incorporate these groupwise constraints into a generalized CCA model, creating a "latent third view" (cluster indicators) that helps align the primary modalities (e.g., image and text).

Optimization

The resulting objective function is non-convex and non-smooth. The authors simplify this by introducing a non-negative orthogonal variable to represent soft group membership and using an alternating minimization strategy. This approach splits the problem into two efficient sub-problems: a generalized ratio-trace problem solved via eigenvalue decomposition and a non-negative problem solved via SVD, ensuring local convergence and computational efficiency.

Experimental Evaluation

The researchers evaluated SPGCM on three standard benchmarks:

  • Wiki Dataset: 2,866 image-text pairs across 10 semantic classes.
  • Pascal VOC 2007: 9,963 image-tag pairs across 20 classes.
  • Deep-Wiki: A version of the Wiki dataset using 4,096-dimensional CNN features for images and 5,000-dimensional TF-IDF features for text to test scalability with high-dimensional data.

Key Results and Insights

SPGCM consistently outperformed other unsupervised baselines like CCA, Partial Least Squares (PLS), and Bilinear Model (BLM) across all tests.

Precision-Scope curves for different retrieval methods. The red line (SPGCM) consistently achieves higher precision than traditional unsupervised methods like CCA and PLS.
  • Superior Accuracy: On the Wiki dataset, SPGCM achieved an average Mean Average Precision (MAP) of 25.38%, significantly higher than CCA's 22.49%.
  • Competitive with Supervised Learning: In some tests, SPGCM performed comparably to state-of-the-art supervised methods (like GMLDA), despite having no access to labels during training. On the "deep-Wiki" dataset, it reached a MAP of 42.4% for image queries.
  • Grouping as Regularization: Groupwise correspondence acts as a "semantic-like" layer, providing more degrees of freedom than strict pairwise matching. This prevents the model from overfitting to noise in individual pairs and makes it more robust to changes in subspace dimensions.

Conclusion and Future Work

SPGCM demonstrates that discovering latent group structures can bridge the gap between unsupervised and supervised cross-modal retrieval. While the method is sensitive to the choice of $K$ (the number of groups) and has a computational complexity of $O(n^2K)$, it provides a robust foundation for learning from unlabeled multi-modal data. Future work may involve exploring non-linear distance metrics or integrating deep learning directly into the subspace learning process to further enhance retrieval precision.

Summary updated

Abstract

Cross-modal learning tries to find various types of heterogeneous data (e.g., image) from a given query (e.g., text). Most cross-modal algorithms heavily rely on semantic labels and benefit from a semantic-preserving aggregation of pairs of heterogeneous data. However, the semantic labels are not readily obtained in many real-world applications. This paper studies the aggregation of these pairs unsupervisedly. Apart from lower pairwise correspondences that force the data from one pair to be close to each other, we propose a novel concept, referred as groupwise correspondences, supposing that each paired heterogeneous data are from an identical latent group. We incorporate this groupwise correspondences into canonical correlation analysis (CCA) model, and seek a latent common subspace where data are naturally clustered into several latent groups. To simplify this nonconvex and nonsmooth problem, we introduce a nonnegative orthogonal variable to represent the soft group membership, then two coupled computationally efficient subproblems (a generalized ratio-trace problem and a non-negative problem) are alternatively minimized to guarantee the proposed algorithm converges locally. Experimental results on two benchmark datasets demonstrate that the proposed unsupervised algorithm even achieves comparable performance to some state-of-the-art supervised cross-modal algorithms.
BibTeX
@inproceedings{liang2016group,
  author = {Jian Liang and Ran He and Zhenan Sun and Tieniu Tan},
  title = {{Group-Invariant Cross-Modal Subspace Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure