TL;DR
This paper introduces a novel cross-modality hashing method (PCCMH) that can effectively learn from datasets where image and text descriptions are not perfectly paired. By leveraging both local smoothness within each modality and partial cross-modal correspondences, PCCMH achieves state-of-the-art performance in cross-modal retrieval tasks, even with incomplete data.
What Problem Does This Paper Address?
Cross-modality hashing aims to map data from different modalities (like images and text) into a common compact binary code space. This allows for efficient similarity search across modalities, enabling tasks like finding images based on text queries or vice-versa. However, existing methods typically require perfectly paired data (e.g., an image and its exact textual description) for training. Acquiring such perfectly aligned datasets is often expensive and difficult, especially when dealing with large-scale, web-crawled data where descriptions can be noisy or incomplete. This paper addresses the challenge of learning effective cross-modality hashing functions when only a portion of the data is reliably corresponded.
Methods
The proposed method, Partial Correspondence Cross-Modal Hashing (PCCMH), tackles the problem by:
- Anchor Graph for Local Smoothness: Instead of building a full similarity matrix (which is computationally expensive for large datasets), PCCMH uses an "anchor graph" approach. This involves selecting a smaller set of representative data points (anchors) and computing similarities between all data points and these anchors. This efficiently captures the local manifold structure within each modality.
- Local Smoothness Preservation: Similar to Spectral Hashing, PCCMH aims to ensure that similar data points in the original feature space are mapped to similar hash codes. This is achieved by minimizing a term that penalizes differences in hash codes for similar data points within each modality.
- Partial Cross-Modal Correspondence: For the data pairs that are corresponded, PCCMH learns to map them to similar binary codes. This is done by minimizing the difference between the hash codes of paired data points from different modalities.
- Optimization Framework: The method unifies these objectives into a single optimization problem. It uses a relaxation technique to make the problem tractable and solves it as an eigenvalue problem to obtain the hashing functions.
Datasets
The paper evaluates PCCMH on two widely-used datasets:
- Wiki: A dataset derived from Wikipedia documents, containing image-text keyword pairs. Images are represented by 128D SIFT BoW features, and text keywords by 10D features from Latent Dirichlet Allocation.
- NUS-WIDE: A large-scale web image dataset from Flickr, comprising 269,648 images associated with 81 semantic concepts. Images are represented by 500D SIFT BoW features, and textual features are derived from PCA on original tag occurrence features.
Key Insights and Ideas
- Leveraging Incomplete Data is Crucial: The core insight is that even partially corresponded data can significantly boost cross-modality hashing performance, making it more practical for real-world scenarios.
- Anchor Graphs for Scalability: Using anchor graphs is an effective way to capture local manifold properties efficiently, making the method scalable to large datasets without the quadratic complexity of full similarity matrices.
- Balancing Local and Cross-Modal Information: The method successfully balances the need to preserve local structure within each modality with the goal of aligning different modalities, even when the alignment is imperfect.
Key Results
- State-of-the-Art Performance: PCCMH consistently outperforms several state-of-the-art cross-modal hashing methods (MMSH, CCA, CRH, MLBE) on both the Wiki and NUS-WIDE datasets.
- Performance Scales with Correspondence: The performance of PCCMH increases as the proportion of corresponded data increases, demonstrating its ability to effectively utilize available information. Notably, performance gains plateau around 50% correspondence on the Wiki dataset, suggesting that a substantial portion of correspondence is sufficient for good results.
- Robustness: The method shows good generalization capabilities, performing well even when the correspondence information is limited.
Limitations and Future Work
- Limitations: The paper acknowledges that the performance improvement is not drastic when the ratio of corresponded data exceeds 50% on the Wiki dataset, possibly due to the dataset's limited diversity. The computational cost of building the anchor graph, while better than full similarity matrices, is still a factor for extremely large datasets.
- Future Work: The authors suggest exploring extensions to handle more than two modalities and investigating more sophisticated ways to model the partial correspondence. Further research into optimizing the anchor selection process could also be beneficial.
Summary updated