Early multimodal models operated on a simple assumption: if one modality is good, two must be better. To extract named entities or analyze sentiment from social media posts, architectures typically threw text tokens and image features into a shared cross-attention mechanism, treating fusion like an unconstrained blender.
However, real-world data quickly broke this assumption. In user-generated content, images are frequently decorative, tangential, or entirely unrelated to the accompanying text. As noted by Lin Sun et al. in RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER, nearly 60% of tweet images do not provide helpful context for the written text. When models are forced to integrate this "visual noise," they begin to hallucinate cross-modal connections, leading to severe performance degradation—often causing multimodal networks to underperform compared to simple, text-only baselines.
To solve this, a major research direction has shifted away from how to fuse modalities toward when to fuse them. By introducing cross-modal relevance gating, modern networks dynamically evaluate the utility of auxiliary data, selectively amplifying helpful context while completely suppressing irrelevant noise.
The Intuition of a Cross-Modal Gate
Instead of allowing visual and textual features to interact immediately, relevance gating introduces a gatekeeper module. This module treats multimodal fusion not as a mandatory integration, but as a conditional filter.
Mathematically, the core intuition relies on computing a scalar or vector relevance score, $g$, based on the semantic alignment between the primary text representation ($x_t$) and the auxiliary visual representation ($x_v$):
$$g = \sigma(W \cdot [x_t, x_v])$$
where $\sigma$ represents an activation function like a sigmoid, bounding the gate value between 0 (completely irrelevant) and 1 (highly relevant). The visual features are then modulated by this gate before fusion:
$$\tilde{x}_v = g \cdot x_v$$
When the image is unrelated to the text, $g$ approaches 0, effectively zeroing out the visual features and preventing their gradients from corrupting the text representations. As argued by Jiquan Wang et al. in RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER, the capacity to actively ignore irrelevant visual context is just as critical to model performance as the ability to integrate helpful context.
[ Text Input ] ------> [ Text Encoder ] -------> [ Text Features ] ---\
| \---> [ Selective Fusion ] ---> Output
v ---> [ Mechanism ]
[ Image Input ] -----> [ Image Encoder ] ------> [ Image Features ] --/
| ^
v |
[ Relevance Gate ] -----/
(Computes Alignment)
Coarse-to-Fine Filtering Strategies
Relevance gating operates across different granularities depending on the complexity of the task, ranging from global image rejection to fine-grained token shielding.
Global Pre-Filtering
The most straightforward approach is an early-exit or coarse-grained filter. Before any deep cross-modal layers are computed, the model runs a lightweight alignment check. In CFSum Coarse-to-Fine Contribution Network for Multimodal Summarization, the authors implement a pre-filtering stage that measures the global consistency between text and image. If the consistency score falls below a predefined threshold, the visual modality is completely excluded from the generation pipeline, saving compute and protecting the text decoder from distraction.
Target and Entity-Level Gating
Often, an image is relevant to a specific entity within a sentence but completely irrelevant to the rest of the text. For fine-grained tasks like sentiment analysis, models must gate features at the entity level.
Jianfei Yu et al. demonstrate this in Targeted Multimodal Sentiment Classification based on Coarse-to-Fine Grained Image-Target Matching by predicting the explicit relevance between an image and a specific target entity. If a tweet discusses a celebrity but the image shows a generic background, the gate down-weights the visual features for that specific entity token.
Similarly, Ming Liu et al. develop a framework in Decompose, Prioritize, and Eliminate: Dynamically Integrating Diverse Representations for Multimodal Named Entity Recognition that dynamically suppresses visual data that fails to align with localized linguistic context, ensuring that decorative images do not corrupt entity extraction.
Shielding Non-Entity Tokens
An alternative perspective focuses on protecting the text rather than filtering the image. In Entity-level Interaction via Heterogeneous Graph for Multimodal Named Entity Recognition, the authors show that excluding non-entity tokens from visual processing entirely is a highly effective way to mitigate noise. By shielding ordinary words (like verbs and prepositions) from image data, the model drastically reduces the rate of "hallucinated" entities.
Soft Guidance vs. Mandatory Input
The architectural implementation of the gate also dictates how robust the model is to noise. Instead of forcing text and vision to meet as equals in a cross-attention layer, recent frameworks treat vision as a "soft guidance" mechanism.
In Good Visual Guidance Make A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction, the authors append visual features to the keys and values of a Transformer's self-attention layers as a "visual prefix" or prompt. Because the visual data acts as an optional prompt rather than a mandatory input token, the model's internal attention heads can naturally learn to bypass the prefix if the relevance gate indicates a low text-image correlation.
This design shift is highly effective. As shown by Xinjing Liu et al. in Multimodal Aspect-Based Sentiment Analysis under Conditional Relation, the ability to filter out irrelevant visual data via a conditional relation detector contributes far more to final model accuracy than the fine-grained localization of objects within a relevant image.
Why It Works: Preventing Negative Knowledge Transfer
The success of relevance gating lies in its ability to prevent negative knowledge transfer—a phenomenon where noisy data from an auxiliary source confuses a model's interpretation of a perfectly clear primary source.
When a text-only model reads a tweet with a typo, it relies on context clues to fix it. If a multimodal model reads the same tweet accompanied by a completely random image, an un-gated cross-attention mechanism will try to find a semantic bridge between the typo and the random image objects. This forces the model to optimize for a relationship that does not exist.
By placing a relevance gate at the frontier of fusion, the architecture preserves the integrity of unimodal representations. The model retains the benefits of multimodal context when text and vision are aligned, while safely falling back to a robust unimodal baseline the moment that alignment breaks down.