Selective Alignment for Noisy Multimodal Contexts

This direction is merged into another direction: web-scale data noise. Its last version stays readable and citable here.

Version 1 of 190 papers99 conceptsfrozen 2026-10-04

In the ideal world of multimodal machine learning, images and text are perfectly synchronized: a photo of a golden retriever accompanies a caption about a golden retriever. In the real world—specifically on social media—this "multimodal assumption" collapses. Users post memes, ironic images, or generic stock photos that have little to do with their written text.

Recent research has pivoted from simple "fusion" (mixing text and image features) to selective alignment. This direction focuses on identifying, filtering, or even synthesizing visual context to ensure that images provide evidence rather than noise.

The Challenge of Decorative and Ironic Noise

The primary hurdle in Multimodal Named Entity Recognition (MNER) and Sentiment Analysis (MABSA) is that a high percentage of images are semantically unrelated to the text. Lin Sun and Jiquan Wang noted in their work on RIVA that nearly 60% of tweet images serve as decorative or emotional markers rather than informative context.

When a model is forced to integrate these unrelated visuals, it suffers from "visual noise." For example, if a tweet mentions "Apple" (the company) but the accompanying image is a decorative photo of a park, a standard model might incorrectly classify the entity as a location. Shuguang Chen and Gustavo Aguilar demonstrated in Can images help recognize entities? that images are often most helpful only in "low-context" scenarios where text is sparse; otherwise, they frequently introduce more confusion than clarity.

Gating and Relation Propagation

To solve this, researchers have introduced "sanity check" mechanisms. Instead of assuming an image is relevant, models now use relation propagation or gating to decide how much weight the visual signal should carry.

In RpBERT, Jiquan Wang and colleagues implemented a mechanism that first determines the text-image relationship. If the image is irrelevant, the model intelligently suppresses the visual features. Similarly, Min Xiao and Junnan Zhu developed DIUSum, which dynamically selects images for summarization based on their utility, effectively learning to ignore "distractions."

Bridging the Gap via Translation

Another powerful intuition in this field is treating the semantic gap as a translation problem. If text and images are "languages" that don't always speak the same thing, we can use one to explain the other.

  1. Image-to-Text: Zaid Khan and Yun Fu proposed EF-CaTrBERT, which translates images into descriptive captions. By treating the image as an "auxiliary sentence," the model can use standard language architectures (like BERT) to find alignments in a purely textual space, which is often less noisy than raw visual feature space.
  2. Text-to-Image (Imagination): When the provided image is too noisy, why not make a better one? Yu Zhuang and Shiliang Sun introduced the ImCo-MMT framework, which uses "Imagination" to generate synthetic images from text. This ensures the visual context is perfectly aligned with the linguistic goal, while a "Contemplation" branch keeps the model grounded in the original source text.

Information Bottlenecks and Selective Attention

For tasks like entity extraction, the noise isn't just in the image—it's in the relationship between specific words and specific pixels. Shiyao Cui and Jiangxia Cao applied the Information Bottleneck principle in MMIB. This mathematical constraint forces the model to discard any information in the image that does not help predict the text, effectively "squeezing out" irrelevant background details.

Similarly, Xiujiao Li and Guanglu Sun developed ESPVR, which uses identified entity spans in text to "point" to specific regions in an image. Instead of looking at the whole noisy image, the model only attends to the visual regions that the text specifically mentions.

Why It Works: The "Sanity Check" Intuition

The success of these methods stems from a shift in philosophy: Text is the anchor, and images are the candidates. By using text-based entity detection or syntactic parsing (as seen in Linlin Zhu's AETS) to simplify the problem first, models can then query the visual modality for specific, corroborating evidence. This hierarchical approach—deciding what to look for before looking at the image—prevents the model from being overwhelmed by the high-dimensional noise inherent in real-world visual data.

Papers in this version

Showing 60 of 90 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_selective_alignment_for_noisy_multimodal_005770d3,
  title = {Selective Alignment for Noisy Multimodal Contexts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/selective-alignment-for-noisy-multimodal-contexts/rdr_005770d3fb84415f92399f168e491de5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_selective_alignment_for_noisy_multimodal_005770d3_v1,
  title = {Selective Alignment for Noisy Multimodal Contexts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/selective-alignment-for-noisy-multimodal-contexts/rdr_005770d3fb84415f92399f168e491de5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}