Seeking Proxy Point via Stable Feature Space for Noisy Correspondence Learning

Yucheng Xie, Songyue Cai, Tao Tong, Ping Hu, Xiaofeng Zhu

Cross-modal retrieval systems, which allow users to search for images using text descriptions and vice versa, have seen significant advancements. However, their performance is often bottlenecked by "Noisy Correspondence"—mismatched image-text pairs commonly found in large-scale web-crawled datasets. A new framework presented at IJCAI-25, Seeking Proxy Point via Stable Feature Space (SPS), introduces a robust method to handle these mismatches by anchoring the learning process in a stable feature space.

The Challenge of Noisy Correspondence

Most cross-modal models rely on contrastive learning to align images and text. This approach assumes that the training pairs are correctly matched. In reality, web data is messy; an image of a "dog" might be paired with a caption about "pancakes." Existing solutions typically try to filter out these pairs or reduce their weight during training. However, this often leads to two major issues:

  1. Information Loss: Discarding noisy pairs wastes potentially valuable visual or textual data.
  2. Feature Instability: Noise causes the mathematical space where features are stored to drift, preventing the model from learning consistent representations.

The SPS Framework

The SPS framework addresses these issues through a three-stage process designed to stabilize the feature space and provide accurate supervision even for noisy samples.

The SPS framework overview, illustrating the transition from feature extraction and loss fitting to stable space construction and proxy point seeking.

1. Fine-grained Data Partitioning

Leveraging the "Memory Effect"—the observation that neural networks learn clean patterns before memorizing noise—SPS uses a Gaussian Mixture Model (GMM) to split the dataset into three subsets: Reliable (clean), Quasi-clean (partially matched), and Noisy (mismatched).

2. Stable Feature Space Construction

To prevent feature drift, the model uses the reliable set to establish a "stable space." It enforces intermodal consistency (ensuring image-to-text and text-to-image transformations match) and intramodal metric consistency (ensuring similar images have similar text feature relationships). This creates a solid anchor for the rest of the learning process.

3. Seeking Proxy Points

This is the core innovation of the paper. For pairs identified as noisy, the model discards the incorrect caption but keeps the image. It then searches the reliable set for the image most similar to the noisy one and uses that image's caption as a "proxy" supervisor. This allows the model to continue learning from the noisy image without being misled by the original, incorrect label.

Illustration of (a) standard noisy correspondence and (b) the SPS approach of seeking proxy points to provide correct supervision for mismatched pairs.

Experimental Results

The researchers evaluated SPS on major benchmarks including Flickr30K, MS-COCO, and the real-world CC152K dataset. The results were particularly striking in high-noise environments:

  • Extreme Robustness: On Flickr30K with 60% noise, SPS achieved a cumulative recall (RSum) of 484.1, while previous state-of-the-art methods like NCR plummeted to 184.6.
  • Real-World Utility: On the CC152K dataset, which contains native web noise, SPS outperformed the backbone model by nearly 10% in total recall.
  • The Necessity of Stability: Ablation studies confirmed that the "Stable Space Constraint" was vital; without it, the accuracy of the proxy search dropped significantly, proving that a stable foundation is required to correct noisy data.

Conclusion

SPS demonstrates that we don't have to "throw the baby out with the bathwater" when dealing with noisy data. By building a stable feature space and seeking proxy supervisors, we can extract reliable knowledge from unreliable sources. Future work may expand this framework into other domains, such as video-audio retrieval, and further optimize the computational efficiency of the proxy seeking process.

Summary updated

Abstract

To meet the growing demand for cross-modal training data, directly collecting multimodal data from the Internet has become prevalent. However, such data inevitably suffer from Noisy Correspondence. Previous works focused on recasting soft labels to mitigate noise's negative impact. We explore a novel perspective to solve this problem: pursuing proxy representation for noisy data to enable reliable feature learning. To this end, we propose a novel framework: Seeking Proxy Point via Stable Feature Space (SPS). This framework employs a fine-grained partitioning strategy to obtain a high-confidence reliable set. By imposing intermodal cross-transformation consistency constraints and intramodal metric consistency constraints, a stable feature space is constructed. Building on this foundation, SPS seeks proxy points for noisy data, enabling even noisy data to be accurately embedded into appropriate positions within the feature space. Combined with partial alignment for partially matched data pairs, SPS ultimately achieves robust learning under Noisy Correspondence. Experiments on three widely used cross-modal datasets demonstrate that SPS significantly outperforms previous methods. Our code is available at https://github.com/C-TeaRanger/SPS.
BibTeX
@inproceedings{xie2025seeking,
  author = {Yucheng Xie and Songyue Cai and Tao Tong and Ping Hu and Xiaofeng Zhu},
  title = {{Seeking Proxy Point via Stable Feature Space for Noisy Correspondence Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/231},
}
Figure