Finding names in short social media posts, such as people, places and organizations, is hard because the text is too short to disambiguate. The attached image often helps. A post reading "Apple" next to a photo of fruit is not about the company. Current multimodal taggers learn a projector, a small network that maps image features into the text model's representation space. They then let a sequence labeler, typically a bidirectional LSTM followed by a conditional random field (BiLSTM-CRF), use those projected features. The trouble is that a freely trained projector can carry anything from the image into the tagger. When the image is a generic selfie unrelated to the text, that freedom turns into noise. The field's main remedy has been to add more learned machinery on top: relevance gates, cross-attention, mixtures of experts.
We propose the opposite. We take the projector's freedom away. The image is allowed to influence the tagger only through a small set of numbers, one per token and entity type. Each number answers a single question: does this image support reading this word as this type, more than an average unrelated image would? We compute these numbers with a frozen contrastive image-text model and add them directly to the CRF's per-token label scores. Only one weight per entity type is learned on top of them. The claim is that this hard constraint keeps most of the visual benefit of learned fusion. It also guarantees, by construction, that an irrelevant image cannot pull the tagger below a text-only model. Learned projectors and gates offer no such guarantee.
Background and Gap
Here a BiLSTM reads a sentence left-to-right and right-to-left to build a context-aware vector per token. The CRF on top scores whole label sequences: it combines per-token label scores ("emissions") with learned scores for each pair of adjacent labels. It then decodes the best legal sequence exactly with the Viterbi algorithm, so invalid transitions such as O→I-PER are ruled out. This structured decoding is what modern multimodal entity taggers keep. Multimodal Named Entity Recognition for Short Social Media Posts adds a global Inception image vector to a BiLSTM-CRF with per-token modality attention. MPCFN fuses BERT text with ResNet region features through a relevance gate and cross-attention before a CRF. MAHE enhances each modality with experts and Mamba blocks before late fusion into a CRF. All three show that images help. All three also inject visual information through high-capacity learned pathways, and the file notes that each remains vulnerable to loosely related images.
Separately, generative vision-language models dispense with the CRF entirely. They map visual tokens into a frozen or tuned language model via a projector, as in Multimodal Few-Shot Learning with Frozen Language Models and MiniGPT-v2. Beyond Unimodal Shortcuts shows that such models applied to entity recognition take shortcuts in both directions. They invent visual evidence from suggestive text, and they predict objects the text never mentions. Fixing this required reinforcement learning, and span boundaries stayed imprecise. These results motivate our proposal: unconstrained cross-modal pathways are where the bias enters.
The key methodological ingredient comes from Interpreting CLIP's Image Representation via Text-Based Decomposition. That paper shows CLIP image representations can be decomposed along text-describable directions. Projecting onto only 60 text directions keeps ImageNet top-1 at 72.77% against 75.25% for the full representation. It also shows that zeroing spurious components sharply improves worst-group robustness. We reuse that insight as the projector itself. The visual signal reaching the tagger is restricted to directions defined by the token's own text and the candidate label.
Core Claim
Fix a frozen CLIP model. It has an image encoder producing patch features $g_p$ and a text encoder $h(\cdot)$, both projected into the shared space by CLIP's own fixed heads. For token $i$ of sentence $x$ and entity type $t$, build a short text query $q_{i,t}$, for example "a photo of [word $i$ with a three-word window], a [type-$t$ description]". The visual evidence for that token-type pair is
$$\Delta_{i,t}(I) = \max_p \cos\big(g_p(I), h(q_{i,t})\big) - \mathbb{E}_{I' \sim \mathcal{R}}\Big[\max_p \cos\big(g_p(I'), h(q_{i,t})\big)\Big].$$
Here $I$ is the post's image and $\mathcal{R}$ is a fixed bank of unrelated reference images. The subtraction removes CLIP's prior that the text alone matches some image. The CRF score of a label sequence $y$ then becomes
$$s(y \mid x, I) = \sum_i \big[E_{i,y_i} + \alpha_{y_i}\,\Delta_{i,y_i}(I)\big] + \sum_i A_{y_{i-1}, y_i}.$$
In this score, $E$ holds the usual emissions from a BERT-BiLSTM, $A$ is the CRF transition matrix, and $\alpha_t$ is one learned scalar per type. The only visual parameters are the $\alpha$'s, at most a few dozen numbers.
Three properties follow. First, the image cannot introduce entities absent from the text, because it only re-scores labels on existing tokens. Second, it cannot shift a token toward a type unrelated to that token's own words, because each evidence term is keyed to the token's text. Third, an unrelated image has zero expected evidence and so, on average, leaves the text-only decision unchanged. Labeling still uses exact Viterbi decoding. What is new relative to MPCFN and the BiLSTM-CRF with modality attention is not that images enter a CRF. It is that they enter only as calibrated, text-anchored per-label potentials rather than through a learned feature projector.
Research Program
We build one text backbone, RoBERTa-base feeding a BiLSTM-CRF, and hold it fixed across all variants. We then compare ways of letting the image in.
The first study is accuracy on standard benchmarks. The second is a robustness study: on the test sets we deliberately pair each post with a random image from another post, and measure how much each fusion method falls below its own text-only counterpart. The third is mechanistic. We compare patch-level maxima against a global image embedding, a fixed type-description query against a learned prompt, and the calibrated difference against raw similarity. The aim is to show which part of the constraint carries the gain.
We also check one known weakness directly. CLIP may encode handles, hashtags and rare names poorly. We stratify results by whether a token is in CLIP's tokenizer vocabulary as a whole word, so that we can report where text-anchored evidence is uninformative rather than discover it as an unexplained shortfall.
Baselines
The naive reference point is the text-only RoBERTa-BiLSTM-CRF trained with the same data, seeds and tuning budget. The file reports no text-only number on Twitter-2015 or Twitter-2017 beyond MPCFN's statement that fusion adds 4–5 F1. On SnapCaptions, the text-only BiLSTM-CRF reaches 48.8 F1 according to Multimodal Named Entity Recognition for Short Social Media Posts. We retrain our own text-only model rather than borrow that number. Under clean images our method must exceed this baseline. Under swapped images it must stay within noise of it, meaning inside the seed-to-seed standard deviation.
The obvious intersection method serves as the key ablation. It is the same backbone with CLIP patch features passed through a learned two-layer projector into the BiLSTM input, in the style of the projectors in MiniGPT-v2. Because only the image pathway differs between this variant and ours, the comparison isolates the effect of constraining the projector. We also run a version with modality dropout during training, which is the cheapest regularization a reviewer would try.
The strongest existing competitor is MPCFN. Its correlation gate, gated injection across layers and cross-attention ahead of a CRF make it the most elaborate learned answer to the same noise problem. It reports 76.74 F1 on Twitter-2015, 87.61 on Twitter-2017 and 82.35 on WuKong. MAHE reports 75.34 and 87.41 on the two Twitter sets. We reimplement an MPCFN-style gate plus cross-attention on our backbone with the same CLIP visual features and the same hyperparameter search budget, and we also quote the published numbers.
Beating MPCFN on clean data is not required. We count the method as successful if it comes within 1 F1 of our MPCFN reimplementation on clean Twitter-2015/2017 while degrading strictly less under image swapping. We count it as clearly superior if it also matches or exceeds that reimplementation on SnapCaptions, where the full multimodal BiLSTM-CRF reports 52.4 F1.
For the generative route we report published numbers only. On Twitter-GMNER, the standalone recognition stage of LLMs as Bridges reaches 85.94 F1. MCR with Qwen2.5-VL-7B in Beyond Unimodal Shortcuts reaches 70.6 F1 on the full grounded task. We evaluate our tagger on the entity-recognition portion of Twitter-GMNER and compare against the 85.94 figure, also reporting parameter counts and latency.
Why not just…?
Why not just concatenate projected image features to the token embeddings? This is exactly our learned-projector ablation. The prior evidence against it is specific. Multimodal Named Entity Recognition for Short Social Media Posts built modality attention because naive concatenation hurt on irrelevant images. Deep Pre-Alignment for VLMs shows that poorly aligned visual inputs waste downstream capacity on translation. If the concatenation variant matches ours under image swapping, the core claim is refuted and we will report it.
Why not just use a learned relevance gate, as MPCFN does? This is the real threat. A gate learned on data where most images are at least loosely relevant may already suppress noise well enough. A gate decides how much of the image to admit. It does not limit what the admitted image can say, so a gate that opens on a misleading image passes unrestricted features into the tagger. The image-swap test settles the question. If the MPCFN-style model's drop under swapping is as small as ours, our constraint adds nothing beyond parameter savings. We would then reframe the work as an efficiency result, not a robustness one.
Why not just add a single image-text similarity score as a sentence-level feature or gate? This is the simplest form of our idea, and we include it as an ablation: one global CLIP similarity scaling all visual influence. It cannot tell which word the image supports or which type it supports. That distinction is exactly what the "Apple"-as-fruit case requires. Our per-token, per-type potentials should win specifically on tokens of ambiguous type. We measure this on a subset of test tokens whose gold type differs from their most frequent training type.
Why not just prompt or fine-tune a multimodal LLM? Beyond Unimodal Shortcuts reports that chain-of-thought MimoVL-7B invents entities absent from the text at a 24.3% rate. Suppressing this took reinforcement learning, and span boundaries stayed weak. Our tagger cannot invent text entities by construction and costs orders of magnitude less. A tuned 7B model may still be more accurate. We would accept that outcome and position our method as the cheap, structurally safe alternative.
Decisive Checks
We run five seeds per configuration on Twitter-2015, Twitter-2017 and SnapCaptions, with the entity-recognition portion of Twitter-GMNER as a secondary set. We report entity-level F1, the F1 change from clean to swapped images relative to each model's text-only counterpart, and accuracy on the ambiguous-type token subset. WuKong is evaluated only if a Chinese CLIP variant is used, and we would flag that as a change of visual encoder.
The proposal stands if two things hold. Text-anchored potentials must recover at least half of the clean-image gain that the learned projector achieves over text-only. And under swapped images they must fall within noise of the text-only model while the learned projector and gate fall measurably below it.
The proposal fails if any of three things happens. The calibrated term contributes no gain over text-only. The global-similarity ablation does as well as the per-token version. Or MPCFN-style gating proves equally robust to swapping. Each failure has a distinct interpretation, and so each is informative.
References
- Multimodal Named Entity Recognition for Short Social Media Posts
- MPCFN: A Multilevel Predictive Cross-Fusion Network for Multimodal Named Entity Recognition in Social Media
- MAHE: a multiscale and hybrid expert-based model for image-text enhanced named entity recognition on social media
- Multimodal Few-Shot Learning with Frozen Language Models
- MiniGPT-v2: Large Language Model as a Unified Interface for Vision-Language Multi-task Learning
- Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognition
- Interpreting CLIP's Image Representation via Text-Based Decomposition
- LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition
- Deep Pre-Alignment for VLMs