Abstract
The presence of noise in acquired data invariably leads to performance degradation in cross-modal matching. Unfortunately, obtaining precise annotations in the multimodal field is expensive, which has prompted some methods to tackle the mismatched data pair issue in cross-modal matching contexts, termed as noisy correspondence. However, most of these existing noisy correspondence methods exhibit the following limitations: a) the problem of self-reinforcing error accumulation, and b) improper handling of noisy data pair. To tackle the two problems, we propose a generalized framework termed as Rank corrElation and noisy Pair hAlf-replacing wIth memoRy (REPAIR), which benefits from maintaining a memory bank for features of matched pairs. Specifically, we calculate the distances between the features in the memory bank and those of the target pair for each respective modality, and use the rank correlation of these two sets of distances to estimate the soft correspondence label of the target pair. Estimating soft correspondence based on memory bank features rather than using a similarity network can avoid the accumulation of errors due to incorrect network identifications. For pairs that are completely mismatched, REPAIR searches the memory bank for the most matching feature to replace one feature of one modality, instead of using the original pair directly or merely discarding the mismatched pair. We conduct experiments on three cross-modal datasets, i.e., Flickr30K, MSCOCO, and CC152K, proving the effectiveness and robustness of our REPAIR on synthetic and real-world noise.
Introduction
Cross-modal matching, such as image-text retrieval, typically relies on the assumption of perfectly aligned data pairs. However, large-scale datasets are often scraped from the web, introducing "noisy correspondence" where images and captions do not match. Existing solutions to this problem often suffer from two major drawbacks. First, they fall into a cycle of self-reinforcing error accumulation: if a similarity network incorrectly identifies a noisy pair as a match, it trains harder on that error, reinforcing the bias. Second, many methods simply discard noisy data, leading to inefficient use of available information.
The REPAIR Framework
To address these limitations, we introduce REPAIR (Rank corrElation and noisy Pair hAlf-replacing wIth memoRy). The framework shifts the focus from raw similarity scores to relational rankings and data recycling.
Rank Correlation (RC)
Instead of relying solely on a similarity network to judge a pair, REPAIR utilizes a memory bank of previously identified "clean" features. For a target image-text pair, the model calculates the distance between the image feature and all features in the memory bank, and does the same for the text feature. If the pair is a true match, the relative ranking of these distances across the memory bank should be highly correlated. By applying Spearman’s rank correlation, the model generates objective soft labels that are independent of the similarity network's internal biases, effectively breaking the feedback loop of error accumulation.
Noisy Pair Half-replacing (NPR)
Rather than discarding pairs identified as noise, REPAIR employs a "half-replacing" strategy. When a pair is deemed mismatched, the model retains one modality (e.g., the text) and searches the memory bank for the most compatible feature from the other modality (e.g., an image feature). This process transforms "trash" data into high-quality synthetic training pairs, maintaining high data volume while improving the overall quality of the training set.
Experimental Results
We evaluated REPAIR across three major benchmarks: Flickr30K, MS-COCO (with synthetic noise levels of 20%, 40%, and 60%), and CC152K (containing real-world web noise).
REPAIR consistently outperformed state-of-the-art (SOTA) methods such as NCR and MSCN. On Flickr30K with 20% noise, REPAIR achieved an rSum (the sum of Recall@1, 5, and 10 for both modalities) of 504.4, a 2.5-point improvement over the previous best. This performance gap widened at higher noise levels, reaching a 5.2-point lead at 60% noise. In real-world scenarios (CC152K), REPAIR achieved an rSum of 369.2, surpassing both NCR (361.2) and MSCN (366.7).
Conclusion and Future Work
REPAIR demonstrates that leveraging rank correlation and memory-based data recycling can significantly mitigate the impact of noisy correspondence. While the framework introduces some computational overhead due to memory bank maintenance—taking approximately 28.1 hours to train on an SGR backbone with a bank size of 4096—the gains in robustness and accuracy are substantial. Future work will focus on better distinguishing between "hard samples" (difficult but correct pairs) and true noise to further refine retrieval performance.
Summary updated