TL;DR
This paper introduces UOT-RCL, a novel framework for cross-modal retrieval that tackles the pervasive issue of noisy labels in real-world datasets. By leveraging Optimal Transport, UOT-RCL simultaneously corrects mislabeled data and bridges the inherent "heterogeneity gap" between different modalities, leading to significantly more robust and accurate retrieval performance.
What Problem Does This Paper Address?
Cross-modal retrieval (CMR) aims to find relevant information across different data types, like matching an image to a text description. Supervised CMR methods, which use labeled data, are particularly effective because they can learn semantic categories. However, obtaining perfectly labeled data is incredibly challenging, especially in the multimodal setting. Real-world datasets, often scraped from the internet, are rife with "noisy labels" – incorrect or ambiguous annotations. Training CMR models with these noisy labels creates two major problems: 1) the model is forced to align semantically unrelated items, and 2) the inherent differences between modalities (the "heterogeneity gap") are exacerbated, leading to poor retrieval. This paper addresses the critical need for CMR methods that are robust to these noisy labels.
Methods
The core of UOT-RCL lies in its innovative use of Optimal Transport (OT) to address both label noise and modality differences within a unified framework.
- Progressive Label Correction with Semantic Alignment (LPLC): This component tackles noisy labels. It frames label correction as a "partial OT problem," where noisy labels are progressively "transported" to more accurate semantic representations. A novel "cross-modal consistent cost function" is introduced to effectively blend information from different modalities and compute a precise transport cost. This process aims to correct mislabeled samples by aligning them with their true semantic categories.
- Bridging Heterogeneous Gap with Relation Alignment (LBHG): To reduce the discrepancy between modalities, this component uses OT to infer semantic-level cross-modal matching. It leverages the relationships between training samples and a set of "confident" cross-modal pairs (identified as having high semantic consistency) to guide the transport plan. This helps align representations from different modalities in a shared semantic space.
Both components are designed to be solved efficiently using the Sinkhorn-Knopp algorithm and are integrated into a single training objective.
Datasets
The authors evaluated UOT-RCL on three widely-used cross-modal retrieval datasets:
- Wikipedia: Contains 2,866 image-text pairs across 10 classes.
- NUS-WIDE: A large-scale multi-label dataset with 269,648 image-text pairs across 81 classes.
- XMediaNet: A multimodal dataset including image, text, video, audio, and 3D models. For this work, they focused on image and text modalities with 200 categories.
The experiments were conducted under various symmetric noise ratios (20%, 40%, 60%, and 80%) to rigorously test the robustness of the proposed method.
Key Insights and Ideas
- Optimal Transport as a Unifying Tool: The paper demonstrates the power of Optimal Transport not just for measuring distances but as a flexible framework to simultaneously tackle complex issues like label noise and modality alignment in CMR.
- Correcting Labels is More Effective than Down-weighting: Instead of simply reducing the influence of potentially noisy samples (as some prior methods do), UOT-RCL actively corrects them. This proactive approach allows for better utilization of the training data.
- Confident Pairs are Crucial: Identifying and leveraging "confident" cross-modal pairs (those with high semantic agreement between modalities) is key to both correcting labels and aligning representations effectively.
- Bridging the Heterogeneity Gap is Essential for Robustness: The paper highlights that the inherent differences between modalities are a significant hurdle, especially when dealing with noisy labels. UOT-RCL's approach to narrowing this gap directly contributes to its improved robustness.
Key Results
- State-of-the-Art Performance: UOT-RCL consistently outperforms existing state-of-the-art methods across all tested datasets and noise levels.
- Significant Robustness Improvement: The method shows particularly strong gains at higher noise ratios (e.g., 80%), demonstrating its ability to handle severely corrupted labels. For instance, on XMediaNet, UOT-RCL surpassed the best baseline by up to 7.4% at high noise levels.
- Ablation Studies Confirm Component Importance: Experiments removing individual components (LPLC, LBHG, or the confident set) showed a degradation in performance, confirming the necessity and contribution of each part of the UOT-RCL framework. The label correction component (LPLC) was found to have the most significant impact.
- Effective Visualization: t-SNE visualizations clearly show that UOT-RCL effectively bridges the heterogeneous gap, aligning visual and textual features in a semantically coherent manner, unlike the original features.
Limitations and Future Work
- Parameter Sensitivity: While the paper shows UOT-RCL is not overly sensitive to the weight factor $\lambda$ (ranging from 0.1 to 0.7), the moving-average factor $\gamma$ requires careful tuning, especially at high noise ratios, to avoid overfitting.
- Computational Cost: While Sinkhorn-Knopp offers efficiency, OT-based methods can still be computationally intensive compared to simpler approaches.
- Future Work: The authors suggest adapting their framework to other CMR methods and exploring its application to more complex multimodal scenarios (e.g., with more than two modalities). They also highlight the potential for further improvements by exploring more sophisticated cost functions and alignment strategies.
Summary updated