Abstract
Cross-modal retrieval typically relies on supervised learning with expensive semantic labels. This paper presents Simultaneous Pairwise and Groupwise Correspondences Maximization (SPGCM), an unsupervised framework that discovers latent semantic structures without manual annotation. By assuming paired data belong to shared latent groups, SPGCM learns a common subspace that captures broader neighborhood relationships. Experimental results on benchmark datasets demonstrate that SPGCM significantly outperforms traditional unsupervised methods and achieves performance competitive with supervised techniques.
Introduction
Cross-modal retrieval—using a text query to find images or vice versa—is a cornerstone of modern search. Most high-performing algorithms are supervised, meaning they require human-annotated semantic labels to learn how different data types relate. Existing unsupervised methods, like Canonical Correlation Analysis (CCA), typically rely on pairwise correspondences: they try to force a specific image and its specific caption to be close in a shared mathematical space. However, this approach is brittle as it ignores the broader "neighborhood" of the data—for example, the fact that all "cat" images and "cat" descriptions should cluster together even without a label. The challenge is to discover these latent semantic structures automatically.
Methodology: The SPGCM Framework
The authors propose a novel framework called SPGCM. Its core innovation is the introduction of groupwise correspondences to supplement traditional pairwise matching.
The Model
Instead of just matching pairs, SPGCM assumes there are $K$ latent groups in the data. It seeks a common subspace where paired heterogeneous data are not only close to each other but also belong to the same latent cluster. They incorporate these groupwise constraints into a generalized CCA model, creating a "latent third view" (cluster indicators) that helps align the primary modalities (e.g., image and text).
Optimization
The resulting objective function is non-convex and non-smooth. The authors simplify this by introducing a non-negative orthogonal variable to represent soft group membership and using an alternating minimization strategy. This approach splits the problem into two efficient sub-problems: a generalized ratio-trace problem solved via eigenvalue decomposition and a non-negative problem solved via SVD, ensuring local convergence and computational efficiency.
Experimental Evaluation
The researchers evaluated SPGCM on three standard benchmarks:
- Wiki Dataset: 2,866 image-text pairs across 10 semantic classes.
- Pascal VOC 2007: 9,963 image-tag pairs across 20 classes.
- Deep-Wiki: A version of the Wiki dataset using 4,096-dimensional CNN features for images and 5,000-dimensional TF-IDF features for text to test scalability with high-dimensional data.
Key Results and Insights
SPGCM consistently outperformed other unsupervised baselines like CCA, Partial Least Squares (PLS), and Bilinear Model (BLM) across all tests.
- Superior Accuracy: On the Wiki dataset, SPGCM achieved an average Mean Average Precision (MAP) of 25.38%, significantly higher than CCA's 22.49%.
- Competitive with Supervised Learning: In some tests, SPGCM performed comparably to state-of-the-art supervised methods (like GMLDA), despite having no access to labels during training. On the "deep-Wiki" dataset, it reached a MAP of 42.4% for image queries.
- Grouping as Regularization: Groupwise correspondence acts as a "semantic-like" layer, providing more degrees of freedom than strict pairwise matching. This prevents the model from overfitting to noise in individual pairs and makes it more robust to changes in subspace dimensions.
Conclusion and Future Work
SPGCM demonstrates that discovering latent group structures can bridge the gap between unsupervised and supervised cross-modal retrieval. While the method is sensitive to the choice of $K$ (the number of groups) and has a computational complexity of $O(n^2K)$, it provides a robust foundation for learning from unlabeled multi-modal data. Future work may involve exploring non-linear distance metrics or integrating deep learning directly into the subspace learning process to further enhance retrieval precision.
Summary updated