Automatically labeled training data for relation extraction (deciding, for example, that a tweet and its photo express "person A lives in place B") comes from distant supervision. The idea is simple: if a knowledge base says A and B are related, any caption or post that mentions both is labeled with that relation. It is cheap, and it is wrong more than half the time. In the multimodal case there is a second source of error, because the attached image may not show either entity. This note proposes one quantity that tells, instance by instance, whether the text and image actually support the automatic label. We train a multimodal masked model, which hides parts of its input and learns to fill them back in, to reconstruct a written description of the distant relation label. We then ask how much the caption and image help with that reconstruction compared with seeing only the two entity names. If the context adds nothing, the label came from the knowledge base alone and the instance is probably noise.
This score does three jobs. It weights the silver (automatically labeled) data during pretraining. It decides which data enters a noisy-to-clean curriculum, in which the model first learns from large noisy data and is then fine-tuned on small human-checked data. It also tests an open question: whether aggressive cross-modal masking falls apart when the image does not contain what the masked text describes. The target is low-resource multimodal relation extraction on MNRE and MORE, where human-annotated data is scarce and the strongest current systems depend either on external search engines or on expensive synthetic image generation.
Core Claim
A cross-modal masked model trained on distantly supervised image–text–relation triples can measure its own label noise, in a single training run, through the gap between reconstructing the relation with context and without it. Using that gap to weight pretraining, and then fine-tuning on gold data, should beat three alternatives on low-resource multimodal relation extraction: unimodal learning-order denoising, global image–text similarity filtering, and pretraining without distant labels. It should beat them most clearly on long-tail relations.
The closest prior work falls into three groups. The first two motivate the method and the third motivates the question. Fine-grained Contrastive Learning for Relation Extraction (FineCL) denoises text-only distant supervision by recording the epoch at which each instance is first predicted correctly. Early-learned instances were 93% accurate and late-learned ones 53%. FineCL uses those weights before gold fine-tuning, but it is text-only, uses ordinary masked language modeling, and needs a full extra training pass just to log epochs. We keep FineCL's staged structure but replace its noise signal with one that is computed online, conditions on the image, and separates text-side from image-side noise. MultiModal Code-Switching (MMCS) builds 773K automatically grounded samples by replacing entity nouns with visual object tokens and reconstructing them. It applies no filtering or staged transfer, and it never masks relation predicates. We mask exactly those predicates and filter by how well they can be reconstructed. Training on Synthetic Data Beats Real Data in Multimodal Relation Extraction (MI²RAGE) generates synthetic images and drops those a teacher classifier cannot label. It is single-stage, supervised, and costly, and its performance degrades past 12 synthetic views. We use real images and a self-supervised filter instead.
Research Program
Building the silver data. We start from image–caption corpora already used for masked multimodal pretraining: CC12M, used by Multimodal Masked Autoencoders Learn Transferable Representations (M3AE), and the 4M Conceptual Captions/SBU/Visual Genome/COCO mix, used by Masked Vision and Language Modeling for Multi-modal Representation Learning (MaskVLM). We link entity mentions in captions to Wikidata, the source FineCL uses for silver labels. Every caption whose linked entity pair has a Wikidata relation mapped to the MNRE or MORE label set becomes a silver instance, consisting of the image, the caption, the entity pair, and the relation. Following Retrieval over Classification (ROC), which retrieves relation labels by matching against natural-language definitions, we write each relation as a short description $y$ (for example, "is a member of the same team as"). This turns the label into text that can be masked and reconstructed.
Pretraining objective. A single-stream transformer in the style of M3AE takes image patches, caption tokens, and the prompt "$e_1$ [description] $e_2$". Caption tokens are masked at a high rate, following M3AE's finding that heavy text masking (50–90%) forces the model to look at the image. Description tokens are always masked. Following UNITER, which found that masking both modalities of the same concept at once removes any anchor, we never mask the image region and the caption span of the same grounded entity together. Grounding boxes come from the MMCS-style detector pipeline.
The noise score. Let $\bar\theta$ be an exponential-moving-average copy of the model's weights, a slowly updated teacher of the kind Semi-Supervised Teacher-Student Architecture for Relation Extraction used to resist memorizing false distant labels. For instance $x$ with caption $T$, image $I$, and description tokens $y_1,\dots,y_n$, the context gain is
$$G(x) = \frac{1}{n}\sum_{t=1}^{n}\Big[\log p_{\bar\theta}(y_t \mid T, I, e_1, e_2) - \log p_{\bar\theta}(y_t \mid e_1, e_2)\Big],$$
which is the average number of nats the context adds to predicting the label beyond what the entity names alone predict. The image-specific gain $G_I(x)$ is the same difference with $T$ present in both terms and $I$ replaced by a null image in the second. A low $G$ flags the classic failure where co-occurrence does not mean the text expresses the relation. A low $G_I$ with a high $G$ flags an image that does not depict the relation.
Each instance's loss is weighted by $w(x)=\sigma\big((G(x)-\tau)/s\big)$, where $\sigma$ is the logistic function, $\tau$ is a running median of $G$, and $s$ is a temperature. The weight applies to the description reconstruction and also to the caption reconstruction when $G_I$ is low. That second use directly addresses the concern that heavy masking over mismatched images teaches the model to hallucinate.
Why the score should not simply memorize the noise. The concern is circular: a model trained on noisy labels might learn to reconstruct them from context as well. Two design choices address it. First, $G$ is computed by the slow teacher, which is updated early in training when, according to FineCL's learning-order audit, networks have mostly fit the clean patterns. Second, the entities-only term absorbs knowledge-base priors, so memorizing a wrong label raises both terms of $G$ roughly equally.
Transfer. Stage 1 is weighted masked pretraining on all silver data. Stage 2 fine-tunes on MNRE or MORE gold data with a retrieval-style head that matches entity-pair representations to relation descriptions. This choice follows the evidence from GE2PE and FineCL that pretraining on noisy data and then fine-tuning on clean data beats mixing the two.
Baselines
The naive reference points are two. The first is the same backbone fine-tuned only on gold MNRE or MORE, with no silver data. The second is the same backbone pretrained with M3AE-style masking on the raw captions and images, with no distant labels at all. The second is essential because it asks whether the distant labels contribute anything beyond generic multimodal pretraining on the same images.
The strongest existing method on full data is ROC, which reports 91.22% F1 on MNRE and 71.97% F1 on MORE. We run ROC's released recipe on our gold splits. For context, Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis reports 84.64% F1 on MNRE and a gain of more than 22% F1 on long-tail relations, but it requires live search-engine calls at test time. We report it as an upper reference for long-tail performance rather than a matched competitor. MI²RAGE reports beating TMR by 3.76 F1 on MNRE-2; the file gives no absolute figure. For low-resource subsets of MNRE and MORE, no reported result was found on Lacuna, so we measure every baseline ourselves on identical 1% and 10% splits. The closest low-resource evidence in the file is text-only: on DocRED at 1% gold data, FineCL reports 33.2 F1 against 29.6 for RoBERTa trained on gold only.
Three ablations isolate the key ingredient. The first is FineCL's learning-order weighting applied to our multimodal backbone and silver data. The second weights instances by a global image–caption similarity score from a contrastive encoder. The third drops weighting entirely and pretrains on the silver data uniformly, as MMCS does. All methods share the backbone, the silver corpus, and a matched pretraining token budget. The budget charges FineCL's extra logging pass against its total. Every method gets the same number of hyperparameter trials, tuned on a gold dev split. The proposal wins if it beats the best of these ablations by more than seed variance (five seeds) at the 1% and 10% gold settings on both benchmarks, and if it improves macro-F1 over the rarest third of relations. Matching ROC on full data while winning in low-resource settings counts as success; losing to uniform silver pretraining counts as failure.
Why not just…?
Why not just apply FineCL's learning-order weighting to a multimodal model? This is the most serious threat, since learning order is already validated as a noise signal. Our score has two concrete advantages. It is computed during the single pretraining run, whereas FineCL needs a separate full pass. And it separates image mismatch ($G_I$) from text mismatch ($G$), which one epoch number cannot. If learning order matches our score both in noise-detection accuracy on the audited set and in downstream F1 under equal compute, we will report that the multimodal gain adds nothing. The head-to-head ablation is designed to settle this question.
Why not just filter with global image–caption similarity? It is cheaper and captures image-side noise. The surveys in the file (Multimodality Representation Learning, Vision-and-Language Pretrained Models) and MMCS all note that global alignment knows an image matches a sentence but not which region matches which word. More importantly, caption–image similarity says nothing about whether the caption expresses the knowledge-base relation, which is where most distant-supervision error sits. We expect similarity filtering to match us on image-mismatch cases and lose on relation-mismatch cases, and the audit is split by noise type to check this.
Why not just generate clean synthetic data like MI²RAGE, or train only on gold? MI²RAGE's own results show performance declining beyond 12 views and depending on costly generation chains. Gold-only training is the naive baseline, and FineCL's DocRED numbers suggest it trails denoised silver pretraining at 1% gold data. Our experiments test whether that holds for multimodal data.
Why not just mix silver and gold in one training run? GE2PE found mixing worse than sequential transfer. We include a mixed run with gold upweighted to confirm this in our setting rather than assume it.
Proposed Experiments
The first decisive check is a noise audit. Annotators hand-label 1,000 silver instances for two things: whether the text supports the relation and whether the image depicts it. On this set we compute the area under the ROC curve of $G$, $G_I$, learning order, and global similarity, reported separately for each noise type. This mirrors FineCL's 93%-versus-53% audit. If $G$ does not beat learning order on text-side noise, the core claim fails regardless of downstream results.
The second check is downstream performance. We report micro-F1 and long-tail macro-F1 on MNRE and MORE at 1%, 10%, and 100% gold data for all baselines and ablations.
The third check is a masking-ratio sweep over caption masking from 15% to 90%, run with and without $G_I$ weighting. This directly tests the tension between M3AE's heavy masking and noisy images. Our prediction is that unweighted performance peaks at a lower ratio and weighting moves the optimum back toward M3AE's range.
The fourth check compares sequential transfer against mixed training, and masking relation descriptions against masking random caption tokens. The second comparison tells us whether reconstructing the label itself, rather than generic masked modeling, carries the effect.
A known risk is the domain gap between CC12M captions and MNRE tweets. We measure it by also building silver data from MMCS's grounded corpus and reporting results from both sources.
References
- Fine-grained Contrastive Learning for Relation Extraction
- MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment
- Training on Synthetic Data Beats Real Data in Multimodal Relation Extraction
- Multimodal Masked Autoencoders Learn Transferable Representations
- Masked Vision and Language Modeling for Multi-modal Representation Learning
- Retrieval over Classification: Integrating Relation Semantics for Multimodal Relation Extraction
- UNITER: UNiversal Image-TExt Representation Learning
- Semi-Supervised Teacher-Student Architecture for Relation Extraction
- GE2PE: Persian End-to-End Grapheme-to-Phoneme Conversion
- Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis
- Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications
- Vision-and-Language Pretrained Models: A Survey