Teaching Background Words to Ignore the Image in Multimodal NER

Multimodal named entity recognition tags people, places, and organizations in short social-media posts using both the text and the attached image. Nearly every word in these posts is background, tagged "outside any entity" (O). Yet the cross-modal attention layers used to fuse the two inputs let every word look at the image. Cross-modal attention is a layer in which each word's representation pulls in information from image regions according to how well they match. The result, documented by Entity-level Interaction via Heterogeneous Graph for Multimodal Named Entity Recognition, is visual interference: ordinary words such as "HURRY" pick up image features and get tagged as entities. The current fix runs a text-only entity detector first and lets only its candidate spans see the image. Because that detector is itself trained on text that is mostly background, it misses rare entities, and once an entity is missed the image can never rescue it.

We propose a softer mechanism. Each word's cross-modal attention gets an extra "null" slot, a learned placeholder that soaks up attention when nothing in the image is relevant. We supervise how much attention goes to this slot. Background words and entities known to have no visual referent are trained to send their attention to the null slot. Entities that do appear in the image are trained to attend to real regions. That supervision is itself extremely imbalanced, so it is trained with class-balanced weighting. The null decision competes inside the same softmax as the image regions, so it depends on both the word and the image. A word the text alone would dismiss can still attend to the image if some region matches it strongly. In two sentences: class imbalance in multimodal NER is a problem inside the attention layer, not only at the output loss. We fix it by giving the attention an explicit, supervised, imbalance-aware option to abstain, instead of hard-gating the image behind a text-only cascade.

Core Claim

Background dominance hurts multimodal sequence labeling in two places. The first is the tagging loss, which the sequence-labeling literature attacks with focal loss (Focal Loss for Punctuation Prediction), Dice loss (IITK@Detox at SemEval-2021 Task 5), or AUC objectives (Low-Resource Named Entity Recognition: Can One-vs-All AUC Maximization Help?). The second is the training signal for the cross-modal attention parameters, which nobody targets. Under ordinary training, roughly nine in ten gradients reaching the word-to-region attention come from background words. Those words have no correct image referent, so the attention is mostly learned from noise.

Our claim is that supervised null attention removes this noise without discarding entities the text-first detector misses. It should match or beat soft-gated and hard-cascaded fusion on entity F1. It should cut the visual-interference rate on O words. It should also recover more of the entities that a text-only tagger misses. Those missed entities are exactly where cascades such as GEI and ESPVR fail by construction.

Mechanism

For word $i$ with query $q_i$ and image regions with keys $k_m$ and values $v_m$, we add a learned null key $k_\varnothing$ and a null value $v_\varnothing$. The attention over regions and the null slot is

$$\alpha_{im} = \frac{\exp(q_i^\top k_m/\sqrt{d})}{\exp(q_i^\top k_\varnothing/\sqrt{d}) + \sum_{m'} \exp(q_i^\top k_{m'}/\sqrt{d})}, \qquad \nu_i = 1 - \sum_m \alpha_{im},$$

where $d$ is the key dimension and $\nu_i$ is the attention mass word $i$ sends to the null slot. The fused output is $\sum_m \alpha_{im} v_m + \nu_i v_\varnothing$.

This borrows the sentinel idea from pointer networks. Adversarial Transfer for Named Entity Boundary Detection with Pointer Networks appends an inactive sentinel token so that long background stretches collapse into one "point to nothing" decision. We move that idea from the output decoder into the fusion layer. The training objective is

$$\mathcal{L} = \mathcal{L}_{\text{CRF}} + \lambda \sum_i w_{a_i}\, \mathrm{BCE}(\nu_i, a_i).$$

Here $\mathcal{L}_{\text{CRF}}$ is the usual conditional-random-field tagging loss. The target $a_i$ is 1 if word $i$ is O or part of an entity labeled ungroundable, and 0 if it is part of an entity with a box in the image. The weight $w_{a_i}$ is an inverse-frequency weight, so the rare $a_i=0$ words dominate how the attention learns. The hyperparameter $\lambda$ trades off the two terms.

The groundability targets come from Grounded Multimodal Named Entity Recognition on Social Media. That paper reports that about 60% of mentioned entities are absent from the image. Without such labels, the target falls back to "abstain if O."

Relation to Prior Work

Three lines of multimodal NER fuse text and image with every word attending to the image. UMT adds a learned visual gate and a text-only span-detection auxiliary task. MPCFN ( MPCFN: A Multilevel Predictive Cross-Fusion Network for Multimodal Named Entity Recognition in Social Media) uses bidirectional predictive cross-fusion. MAHE ( MAHE: a multiscale and hybrid expert-based model for image-text enhanced named entity recognition on social media) is a third model in the same family. These gates are learned only from the tagging loss, so they inherit the background-dominated gradient. UMT's gate is also computed without competing against the image regions. We differ by supervising the abstention directly and by placing it inside the region softmax.

GEI and ESPVR motivate the proposal: both found that background words attending to the image cause errors. They also constrain it, since their hard exclusion trades interference for error propagation from the text-first detector. That error propagation is the weakness GEI itself names, and H-Index's 3.96-point gain over pipelines on Twitter-GMNER corroborates it.

Cross-modal Background Suppression for Audio-Visual Event Localization supplies the conceptual precedent: one modality can down-weight the other's features where it detects background. It does this with a fixed scaling hyperparameter on 1-second video segments. We make the suppression learned, supervised, and word-level. CMAF-Net pairs attention fusion with a class-balanced focal loss under 99:1 imbalance. It shows the combination matters, but for image-level classification with the rebalancing at the output. We apply the rebalancing to the attention itself. Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance studies imbalance in multimodal entity labeling, but on pre-segmented entities, so background words never arise there.

Research Program

We will build the null-attention layer on a shared backbone: a BERT-family text encoder, a grid-based vision encoder as in ESPVR (avoiding closed-vocabulary detectors), and a CRF head. The layer replaces the text-to-image cross-attention used in UMT-style models. Training uses the combined objective, with groundability targets where Twitter-GMNER provides them.

GMNER is built on Twitter-2015 and Twitter-2017 tweets, so we will check split membership tweet by tweet. Groundability labels will only be drawn from tweets in the MNER training splits, to avoid test leakage.

The analysis has three parts. First, we measure what fraction of the attention-parameter gradient comes from O words before and after supervision. This is the mechanism claim. Second, we ask whether $\nu_i$ is calibrated against groundability on held-out GMNER data. Third, we ask whether entities missed by a text-only tagger receive low $\nu_i$ when their image region is salient.

Baselines

The naive reference point is the text-only BERT-CRF on the same splits. MPCFN reports gains of "around 4-5% over text-only baselines" on Twitter-2015. Any multimodal method that fails to clear text-only by a clear margin is not using the image. The second naive point is the same backbone with plain, unsupervised cross-attention over all words. It isolates whether fusion helps at all before any background handling.

The strongest reported competitor is MPCFN, which reports 76.74% F1 on Twitter-2015 and 87.61% on Twitter-2017 (MPCFN). It is the strongest because it is the best dense-fusion number in the file and shows that bidirectional cross-attention beats ablated fusion. We will reimplement its fusion module on our shared backbone, with the same image encoder, epochs, and number of hyperparameter search trials as our method. We will report both its published numbers and our reimplementation. Beating it means a higher mean entity F1 over five seeds on both Twitter datasets, with the gap exceeding seed standard deviation. A tie in F1 combined with a significantly lower interference rate would count as a partial win.

The strongest cascade competitor is GEI, which reports 74.43 on Twitter-2015 and 86.75 on Twitter-2017. We will reimplement its key ingredient, image access only for text-detected candidate spans, on the same backbone. We also include a high-recall variant whose detector threshold is lowered until its candidate recall matches the abstention head's recall on entity words.

For grounding, H-Index reports 56.41% F1 on Twitter-GMNER and is the reference for whether the null slot's groundability predictions are competitive. We use it as a reference only, since its generative decoder differs from our tagger.

Three ablations isolate the key ingredient. The first removes the abstention supervision but keeps the null slot (unsupervised sentinel). The second keeps the supervision but replaces the in-softmax null slot with a scalar sigmoid gate computed from the text alone, i.e. a supervised UMT-style gate. The third keeps supervision but drops the class-balanced weights $w$.

Why not just…?

Why not just put focal or Dice loss on an existing dense-fusion model? This changes how much each word's tagging error counts. It does not change what background words absorb from the image during the forward pass. GEI's "HURRY" example shows that the corruption happens before the tagger. Still, focal loss could plausibly close much of the gap. We will therefore run MPCFN-style fusion with focal loss, tuned over $\gamma$ with the same trial budget. We will also stack focal loss onto our method to test whether the two effects add up. If focal-loss dense fusion matches us on F1 and on interference rate, the attention-level claim fails.

Why not just use UMT's learned visual gate, which already lets words down-weight the image? This is the most serious alternative. It is why the scalar-gate-with-supervision ablation exists. If a supervised scalar gate equals the null slot, the contribution is the imbalance-aware supervision rather than the sentinel architecture, and we will say so. We expect the in-softmax slot to win specifically on text-missed entities, because its abstention decision sees the query-region match while a text-derived gate does not.

Why not just lower the threshold of GEI's text-first detector so fewer entities are dropped? Lowering the threshold passes more background spans to the image and reintroduces interference. The matched-recall cascade baseline measures this tradeoff directly. Our method must sit above the cascade's precision-recall curve at matched recall, not merely at one operating point.

Why not just prompt a multimodal large language model? The file contains no MNER numbers for such models. The only evidence is on dialogue segmentation, where One-Dimensional Object Detection for Streaming Text Segmentation of Meeting Dialogue reports GPT-4o at 38.21 versus 59.10 for a specialized model. That task is different, so we treat zero-shot prompting as an optional reference rather than settled evidence.

Proposed Experiments

We evaluate on Twitter-2015 and Twitter-2017 for entity F1 and per-type F1, with attention to the rare MISC type. We evaluate on Twitter-GMNER for groundability accuracy of $\nu_i$ and for grounded F1 against H-Index's 56.41. We evaluate on SnapCaptions, where context is extremely sparse, as a transfer check against the reported 52.4% multimodal and 48.8% text-only F1 (Multimodal Named Entity Recognition for Short Social Media Posts).

We also report two diagnostics. The visual-interference rate is the share of O words that are tagged correctly by the text-only model but become false positives in the multimodal model. The rescue rate is the share of gold entities missed by the text-only tagger that the multimodal model recovers.

The decisive checks are three. The rescue rate must be clearly above the matched-recall cascade's. The interference rate must be clearly below dense MPCFN-style fusion's, at equal or better F1. The supervised null slot must beat the supervised scalar gate on rescue rate. If the first check fails, the method is a soft cascade with no advantage. If the second fails, the background problem is not being solved inside attention. If the third fails, we report the result as a supervision finding rather than an architectural one.

References

  1. Entity-level Interaction via Heterogeneous Graph for Multimodal Named Entity Recognition
  2. Focal Loss for Punctuation Prediction
  3. IITK@Detox at SemEval-2021 Task 5: Semi-Supervised Learning and Dice Loss for Toxic Spans Detection
  4. Low-Resource Named Entity Recognition: Can One-vs-All AUC Maximization Help?
  5. ESPVR: Entity Spans Position Visual Regions for Multimodal Named Entity Recognition
  6. Adversarial Transfer for Named Entity Boundary Detection with Pointer Networks
  7. Grounded Multimodal Named Entity Recognition on Social Media
  8. Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer
  9. MPCFN: A Multilevel Predictive Cross-Fusion Network for Multimodal Named Entity Recognition in Social Media
  10. MPCFN: A Multilevel Predictive Cross-Fusion Network for Multimodal Named Entity Recognition in Social Media
  11. MAHE: a multiscale and hybrid expert-based model for image-text enhanced named entity recognition on social media
  12. MAHE: a multiscale and hybrid expert-based model for image-text enhanced named entity recognition on social media
  13. Cross-modal Background Suppression for Audio-Visual Event Localization
  14. CMAF-Net: cross-modal attention fusion with information-theoretic regularization for imbalanced breast cancer histopathology
  15. Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance
  16. One-Dimensional Object Detection for Streaming Text Segmentation of Meeting Dialogue
  17. Multimodal Named Entity Recognition for Short Social Media Posts