Robust Multimodal Alignment under Noisy Correspondence

This direction is split into other directions: noisy correspondence. Its last version stays readable and citable here.

Version 1 of 180 papers86 conceptsfrozen 2026-10-04

In the traditional supervised learning paradigm, we assume that our labels are the ground truth. If an image is labeled "dog," the model learns "dog-ness." When labels are wrong—a "noisy label" problem—the model might learn that a cat is a dog. However, as we scale to massive, web-scraped multimodal datasets, a more insidious problem emerges: Noisy Correspondence (NC).

In noisy correspondence, the individual data points are often correct, but the link between them is broken. An image of a sunset might be paired with a caption about a pancake recipe. As Haochen Han, Minnan Luo, and colleagues note in Noisy Correspondence Learning with Meta Similarity Correction, this forces unrelated data representations to converge in a shared vector space, poisoning the model’s semantic understanding.

The Geometry of Mismatch

Why is noisy correspondence harder to fix than simple label noise? In single-modality classification, noise is categorical. In multimodal alignment, noise is high-dimensional and relational.

Standard contrastive learning (like CLIP) works by pulling "positive" pairs together and pushing "negative" pairs apart. When a pair is mismatched (a false positive), the model aggressively pulls a "sunset" vector toward a "pancake" vector. Because these concepts are semantically distant, this "force" is massive, distorting the entire embedding space.

To solve this, researchers have moved away from "blind" alignment toward frameworks that treat the correspondence itself as something to be predicted, verified, or smoothed.

Identifying the "Toxicity" of Pairs

One intuitive approach is to build a "judge" that evaluates the reliability of each pair before the main model learns from it. Hao Li introduced the Negative Pre-aware Cross-modal (NPC) framework (Negative Pre-aware for Noisy Cross-Modal Matching), which measures a sample's "toxicity." By observing how a training pair affects the model's performance on a small set of known clean data, the system can adaptively down-weight harmful mismatches.

Similarly, Haochen Han and Qinghua Zheng proposed the Meta Similarity Correction Network (MSCN). This uses a meta-learning loop to assign reliable similarity scores to training pairs, effectively "purifying" the dataset during training. Instead of a binary "match or no match," the model learns a continuous degree of correspondence.

Leveraging Intra-Modal Stability

If the bridge between an image and a caption is broken, can we use the "neighbors" within each modality to fix it? This is the core intuition behind IN²R (Intra-Modal Neighbors Never Lie) by Yang Liu and Jiancheng Lv.

The idea is that while the inter-modal link (image-to-text) might be noisy, the intra-modal structure (image-to-image) is usually stable. If an image is surrounded by other images of sunsets, but its paired text is about pancakes, the model can look at the text paired with those neighboring sunset images to synthesize a "soft prototype." This provides a more accurate supervision signal than the original, noisy caption.

Zihua Zhao and Bo Han extend this in Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning. They argue that a true match should preserve the geometric structure across modalities: if two images are similar, their corresponding texts should also be similar. Any pair that violates this "structure consistency" is likely noise.

Disentangling Signal from Noise

Sometimes, the noise isn't a total mismatch but a partial one—a caption that only describes the background, or an image with multiple objects where only one is mentioned.

In Disentangled Noisy Correspondence Learning, Minnan Luo and Zhuohang Dang propose DisNCL. This framework uses information theory to separate "shared" information (the actual correspondence) from "modality-specific" noise. By explicitly modeling the noise as a separate component, the model can focus its alignment efforts on the features that actually bridge the two modalities.

Global Alignment via Optimal Transport

Rather than looking at pairs in isolation, some researchers treat alignment as a global assignment problem. Manh Luong and Dinh Phung explore this in Revisiting Deep Audio-Text Retrieval Through the Lens of Transportation.

By using Optimal Transport (OT), the model finds the most efficient way to map a distribution of audio samples to a distribution of text samples. Because OT considers the entire batch at once, it is inherently more robust to individual outliers. If one pair is a total mismatch, the "cost" of forcing that alignment becomes too high, and the OT plan naturally seeks a more plausible global configuration.

Why This Matters

As we move toward "Foundation Models" trained on the entire internet, we can no longer afford the luxury of clean, human-curated data. The research direction represented by these papers—from Robust Dual Embedding (RDE) by Qin Yang to CANDY by Ruiming Guo—shifts the burden of data cleaning from humans to the learning algorithm itself. By treating correspondence as a probabilistic and structural challenge rather than a given fact, these models achieve high performance even when the majority of their training data is technically "wrong."

Papers in this version

Showing 60 of 80 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_robust_multimodal_alignment_under_noisy__1f997e24,
  title = {Robust Multimodal Alignment under Noisy Correspondence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-multimodal-alignment-under-noisy-correspondence/rdr_1f997e243348496c81bd83b324ad572b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_robust_multimodal_alignment_under_noisy__1f997e24_v1,
  title = {Robust Multimodal Alignment under Noisy Correspondence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-multimodal-alignment-under-noisy-correspondence/rdr_1f997e243348496c81bd83b324ad572b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}