Systems that ground text in images, meaning they point to the box an image region a phrase refers to, nearly always assume the phrase describes something that is actually in the picture. Ask such a model for "the white cup" in a room with no cup and it will confidently box the white plate. The usual fix is to show the model negative examples: phrases describing absent things, paired with an explicit "nothing here" answer. Those negatives are expensive to annotate. The cheap alternative is to generate them automatically, by having a language model propose plausible missing objects and having an off-the-shelf detector confirm they are absent. That cheap route has a hidden flaw. The confirming detector has blind spots, so some "absent" negatives describe objects that are really present, and the grounding model gets punished for being right.
This proposal treats the truth of every training signal as unknown, and estimates it with the model. A grounding model is trained from image–caption pairs only, with no boxes. Its output space includes a "not in this image" option alongside the candidate regions. Positive phrases (from the caption) and proposed negative phrases (from a language model, scored by a frozen detector) are both treated as noisy evidence about whether the phrase's referent is present. The model alternates two steps. In the first, it infers, for each phrase, a posterior over "which region, or none". In the second, it retrains on those soft targets and re-estimates how often each noise source lies. The bet is that a model allowed to say "nothing here" can learn to reject absent targets from captions alone. It should also reach better false-alarm behaviour than trusting a filtered negative set, without paying the drop in ordinary grounding accuracy that heavy negative training is known to cause.
Core Claim
Weakly supervised grounding with expectation–maximization (EM, an algorithm that alternates between guessing hidden variables and refitting parameters to those guesses) already works well. Contrastive Learning with Expectation-Maximization for Weakly Supervised Phrase Grounding treats the phrase-to-region alignment as hidden and reaches 63.05% top-1 accuracy on Flickr30K Entities from captions alone. Its hidden variable is a distribution over the regions of the paired image that sums to one, so it can never say "absent." Its only negatives are regions from other images in the batch.
Separately, absence-aware training works when negatives are trustworthy. Vision-Language Models Do Not Understand Negation generates absent-object captions with an LLM, verifies them with OWL-ViT, and gains 10% recall on negated queries and 28% on negation multiple-choice. However, it treats the detector's verdict as ground truth, and the research file flags exactly this reliance on oracle detectors as an open problem.
Our claim is that both limitations disappear if the alignment variable gets a null state, and if both the "present" claim from the caption and the "absent" claim from the negative proposer are modeled as noisy observations of that state, with reliabilities estimated inside the EM loop. What is new is not "add negatives to EM." It is that the absence label itself is latent, and that the external verifier becomes one noisy annotator whose trustworthiness is learned rather than a filter that is obeyed.
Research Program
Setup. For an image $I$ with region proposals $1..K$ and a phrase $p$, the grounding network with parameters $\theta$ outputs $p_\theta(z \mid I, p)$ over $z \in \{0, 1, \dots, K\}$, where $z=0$ means "the referent is not in the image." Each phrase carries an observed source tag $s$. The tag is "caption" if the phrase was extracted from the image's own caption, and "proposed-absent" if an LLM suggested it as a plausible missing object or attribute. Each phrase also carries a detector score $d$ from a frozen open-vocabulary detector. Let $a = \mathbb{1}[z \neq 0]$ denote presence.
We posit a small noise model. The source tag lies at rate $\epsilon_s$: captions sometimes mention invisible things ("waiting for his wife"), and proposals are sometimes actually present. The detector score follows class-conditional densities $f(d \mid a)$, fit as two Beta distributions.
E-step. The posterior over alignment-or-absence combines the network's belief with the noisy observations:
$$q(z) \propto p_\theta(z \mid I, p)\; p(s \mid a(z); \epsilon)\; f(d \mid a(z)),$$
where $p(s \mid a; \epsilon)$ is the source-flip model ($1-\epsilon_s$ if the tag agrees with $a$, $\epsilon_s$ otherwise). The probability mass on regions is shared exactly as in Chen et al. The new piece is the mass on $z=0$.
M-step. We train $\theta$ with cross-entropy to $q$. We keep Chen et al.'s cross-image contrastive term so that positive alignment does not regress. We then re-estimate $\epsilon$ and the detector densities from expected counts under $q$.
This is a Dawid–Skene-style treatment of annotator reliability, in which caption, proposer and detector are three annotators, wrapped around a deep grounding model. Pseudo-labeling has already been formalized as approximate EM by Expectation Maximization Pseudo Labels. We borrow that framing, together with its lesson that hand-fixed confidence cutoffs are worse than estimated ones. That paper learns a threshold for pixel labels. We learn the reliability of absence evidence.
Degenerate solutions. Two failure modes need guarding, and the program studies both explicitly.
The first is collapse to "everything absent," which mirrors the affirmation-collapse warning in the negation paper. It is prevented by constraining $\epsilon_s < 0.5$ per source, which keeps the problem identifiable, and by warm-starting $\theta$ from a Chen-style model.
The second is confirmation bias, where early confident mistakes get locked in. Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models reports this kind of saturation as self-training iterations grow. We address it by keeping the E-step soft rather than argmax and by tracking estimated noise rates against audited ones across rounds.
Hard negatives. Negatives come in three tiers. Cross-paired phrases from unrelated captions are easy. LLM-proposed co-occurring objects ("cup" in a kitchen) are medium. Attribute or relation swaps of caption phrases ("yellow helmet" when the caption says red), in the spirit of DesCo's confusable negatives, are hard. DesCo: Learning Object Recognition with Rich Language Descriptions showed that such confusable negatives are what break bag-of-words shortcuts. Each tier gets its own $\epsilon$, because attribute swaps are exactly where both the proposer and the detector are least reliable.
Training data. Images and captions come from COCO and Flickr30K. No boxes are used in training. COCO's category annotations are used only as an audit, to measure the true noise rate of category-level negatives. They are never used as a training signal.
Baselines
The naive reference point is the Chen et al. weakly supervised EM grounder retrained on our data and proposals. It has no null state, so every phrase gets a region. Its reported Flickr30K Entities top-1 is 63.05% and its RefCOCO+ accuracy is 59.51% on TestA and 43.46% on TestB, all from Contrastive Learning with Expectation-Maximization for Weakly Supervised Phrase Grounding. On negative-containing benchmarks we expect it to fail by construction. It defines the positive-grounding accuracy we must not lose: beating it means rejection metrics improve while Flickr30K and RefCOCO+ accuracy stays within one point of our reproduction.
The strongest competitor on this exact question is the trust-the-filter pipeline. The same architecture and null state are trained with the same LLM-proposed negatives, but a negative is kept only if the frozen detector's score falls below a threshold. Kept negatives get a hard "absent" target, and caption phrases get a hard "present" target. This is the recipe of Vision-Language Models Do Not Understand Negation transplanted to grounding. It gets the same data, the same detector and the same GPU-hours. Its threshold is tuned on the same validation budget we use for our two hyperparameters (the $\epsilon$ cap and the contrastive weight). Beating it means a higher false-alarm discovery rate and OmniLabel AP at equal or better positive accuracy, with the gap widening as audited negative noise rises.
The supervised reference is Iterative Robust Visual Grounding with Masked Reference based Centerpoint Supervision (IR-VG), which trains an explicit existence head on gold boxes and curated negatives. It reports 82.60% on RefCOCO testB and a 25% relative gain in false-alarm discovery rate over prior methods on RefCOCOg_F and ReferItGame_F. The file gives no absolute $R_{fad}$ or $R_{mix}$ value, so there is no reported result found on Lacuna for an absolute IR-VG false-alarm rate. We will evaluate its released predictions if available and otherwise report it only as relative context. It is an upper bound to approach, not one we expect to beat.
On OmniLabel, the external reference points are DesCo-FIBER at 29.3 AP and GLIP-L falling from 33.2 to 21.2 AP once negative descriptions are included, both reported in OmniLabel: A Challenging Benchmark for Language-Based Object Detection and the DesCo paper. These models are trained with boxes, so they are context rather than direct competitors.
Three ablations isolate the key ingredient. In the first, $\epsilon$ is frozen at its initial values and never re-estimated. In the second, the detector is removed as an annotator, leaving only the source tags. In the third, a hard E-step replaces the soft one. If freezing $\epsilon$ matches the full method, the "latent absence label" story is wrong and the gain is just from the null state.
Why not just…?
Why not just pair images with random captions from other images and train the null state on those? Cross-pairing is what All You May Need for VQA are Image Captions uses to teach zero-counts, and it is nearly free. But such negatives are semantically distant. OmniLabel's drop for GLIP-L, from 33.2 to 21.2 AP once in-image plausible negatives appear, shows that distant negatives do not transfer to plausible ones. We include cross-pairing as the easy tier and report a cross-pairing-only variant. If it matches the full method on the attribute-swap split of RefCOCOg_F, we will say so. Our expectation is that it helps on irrelevant-description negatives and fails on attribute swaps.
Why not just filter the negatives with OWL-ViT and trust them? This is the real threat. If the detector's false-absence rate on our proposals is low, then EM has nothing to correct and the filtered pipeline will tie. We do not argue around this; we measure it. Using COCO category annotations, we audit the actual contamination of filtered negatives, and we run a controlled contamination sweep (described below). If the filtered baseline ties at realistic noise levels, the contribution narrows to robustness at high noise, which matters mainly for attribute and relation negatives where detectors are weakest.
Why not just read absence from the model's internal "absent" neurons at inference? Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens finds neurons that fire for ungrounded tokens and lifts LLaVA-v1.5 "No"-accuracy from 48.0% to 77.5% with a linear probe. That probe is trained on labeled contrastive pairs, applies to generative chat models rather than box-level grounders, and corrects outputs post hoc without improving localization. A version of this probe could plug into our E-step as a fourth noisy annotator. We leave that as an extension rather than claiming to supersede it.
Why not just use a frontier model as judge, or a bigger model? The negation paper reports that scale and newer contrastive objectives do not remove affirmation bias. Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach shows that cheap self-generated negatives with a frozen CLIP check cut Object HalBench hallucination from 24.6 to 5.1 without a frontier judge. Its frozen check is the same kind of static oracle we propose to make estimable.
Proposed Experiments
Main comparison. Every method in Baselines is trained on identical COCO and Flickr30K image–caption data with identical proposals. Positive grounding is evaluated on Flickr30K Entities and RefCOCO+ (top-1 at IoU ≥ 0.5). Rejection is evaluated on RefCOCOg_F and ReferItGame_F ($R_{fad}$, $R_{mix}$), split by irrelevant-description versus attribute-swap negatives. OmniLabel AP is also reported, using presence posterior times region posterior as the box score. That last result probes free-form zero-object descriptions, but proposal-based grounding is not a detector, so it is treated as secondary.
First decisive check: does the null state cost positive accuracy? If Flickr30K top-1 drops more than one point below the Chen reproduction, we have reproduced the affirmation-collapse trade-off rather than escaped it.
Second decisive check: does EM correct the verifier? We take COCO-audited clean category negatives and deliberately replace 0%, 10%, 20% and 40% of them with phrases naming objects that COCO marks as present. We then train both the filtered baseline and our method. Our method should hold $R_{fad}$ and positive accuracy roughly flat while the baseline degrades. Its estimated $\epsilon$ for that tier should track the injected rate. If $\epsilon$ does not track it, the method is not doing what we claim, whatever the headline numbers say.
Third decisive check: confirmation bias over rounds. We plot audited noise, estimated noise and benchmark metrics over EM rounds. The self-improving NER paper predicts a plateau and then a decline for naive loops. A soft E-step with learned reliabilities should plateau without the decline.
Generality, as a lighter follow-up. The same latent-NIL construction applies to unsupervised entity linking, where Boosting Entity Linking Performance by Leveraging Unlabeled Documents explicitly leaves mentions absent from the knowledge base unsolved. We would test it there only after the grounding result holds.
References
- Contrastive Learning with Expectation-Maximization for Weakly Supervised Phrase Grounding
- Vision-Language Models Do Not Understand Negation
- Expectation Maximization Pseudo Labels
- Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models
- DesCo: Learning Object Recognition with Rich Language Descriptions
- Iterative Robust Visual Grounding with Masked Reference based Centerpoint Supervision
- OmniLabel: A Challenging Benchmark for Language-Based Object Detection
- All You May Need for VQA are Image Captions
- Unveiling the Response of Large Vision-Language Models to Visually Absent Tokens
- Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach
- Boosting Entity Linking Performance by Leveraging Unlabeled Documents