When a vision-language model has to name what is in a photo out of millions of possible Wikipedia entities, two things go wrong. The first is a geometric gap. Image and text embeddings from contrastive models such as CLIP (a model trained to pull matching image–caption pairs together) sit in separate regions of the shared space, so matching an image directly against entity names is a cross-modal comparison with a built-in offset. The second is hallucination. A generative model can write out a plausible entity name, but the name may not exist.
This proposal joins the two failures. When a generated name checks out against a real entity catalog, the query can be moved to the text side of the gap and matched text-to-text, where no gap exists. How much the system relies on that text-side route is set per example by how strongly the name validated, not by the model's own confidence. When validation fails, which should happen mostly on rare and unseen entities, the system falls back to the direct image-to-text comparison, corrected for the gap.
Put in two sentences: lexical validation, meaning a check of generated strings against a dictionary or entity list, is usually a final filter that throws away bad outputs. Here it becomes the signal that decides, for each input, whether to cross the image–text gap through a verified word or through geometry.
Core Claim
In open-domain visual entity recognition, a fusion weight computed from discrete validation evidence will outperform three alternatives on the unseen-entity split: the best fixed interpolation of generative and dense routes, a learned gate over continuous features, and a gate driven by the generator's own confidence. The validation evidence is whether beam candidates resolve to catalog entries, how cleanly they resolve, and whether several candidates agree. The claim fails if the generator's log-probability carries the same information. That comparison is the central experiment.
Where This Sits
Three lines of prior work motivate the idea without doing it.
Lexical validation has repeatedly delivered large gains as a post-hoc filter. Filtering character-level translation beams against the Oxford English Dictionary raised historical spelling normalization from 39.8% to 78% on 16th-century letters (Revisiting NMT for Normalization of Early English Letters). A lexicon filter cut word error rate by about 10 points in Japanese loanword back-transliteration (Back-Transliteration of English Loanwords in Japanese). Checking candidate keywords collectively against the corpus held hallucination to about 1% in domain query generation (tRAG). In all three, validation is a hard accept or reject step that the rest of the system never sees. We reuse the collective-verification idea from tRAG, where agreement across candidates counts as evidence, but we turn its output into a continuous weight.
The OVEN benchmark paper (Open-domain Visual Entity Recognition) is the closest work. It grounds PaLI's free-form outputs to Wikipedia titles with BM25 keyword search, but only as offline post-processing, and it never fuses the result with a dense retriever. Its numbers also expose the regime we target. PaLI-17B reaches 30.6% on seen entities but only 12.4% on unseen ones, and 31.6% on a gold subset where humans reach 77.7%. We read the seen/unseen drop as a sign that the generative route's validation evidence should become weaker exactly where that route is unreliable. That is the property a validation-driven gate needs.
Gap mitigation supplies the dense route and also constrains its design. Mind the Gap explains the gap as a product of how the two encoders are built and of the contrastive loss itself. GR-CLIP (Closing the Modality Gap for Mixed Modality Search) removes it by subtracting each modality's mean embedding, and it evaluates on MixBench subsets built from OVEN and VisualNews. AlignCLIP argues that such global shifts distort local neighborhoods. ModGap shows that gap size matters little for rank-based retrieval but a lot for distance-based grouping.
These results shape our design in two ways. First, we do not try to close the gap better. We route around it: text-to-text comparison of a validated name against entity titles never crosses the gap, so AlignCLIP's distortion concern does not apply to that route. Second, ModGap's point that ranking is gap-insensitive warns that fusion across the two routes must work on calibrated scores, not raw cosines. The adaptive-weighting literature, including AMIF's divergence-conditioned attention (AMIF) and the bi-level loss weighting in Facilitating Multimodal Classification via Dynamically Learning Modality Gap, always computes its gates from continuous internal features. We test whether an external discrete signal does better.
Research Program
The system has two routes that score the same candidate pool of entities.
The dense route embeds the image and the query text with a frozen CLIP model. It applies GR-CLIP mean-centering, using calibration means computed on held-out data, and scores each entity by cosine similarity to the centered embedding of the entity's name and, where available, its image.
The symbolic route runs an open generative vision-language model to produce $k$ beam candidates. Each candidate is resolved against the full OVEN entity catalog with BM25. The resolved titles are then embedded with the same CLIP text encoder, and entities are scored by text-to-text similarity to those anchors. A candidate's share of the anchor is proportional to its validation strength.
The final score for entity $e$ given input $x$ is
$$s(e \mid x) = w(v_x)\, s_{\text{sym}}(e \mid x) + \big(1 - w(v_x)\big)\, s_{\text{dense}}(e \mid x),$$
where $s_{\text{sym}}$ and $s_{\text{dense}}$ are the two route scores after per-route temperature calibration on validation data. The vector $v_x$ holds only validation features: whether the top candidate matches a title exactly, the BM25 margin between the first and second resolved entities, the fraction of the $k$ beams that resolve to the same entity, and the catalog-side frequency of the resolved entity. The function $w$ is a small logistic model that outputs a weight between 0 and 1. It is trained on a held-out slice of the OVEN training split to maximize accuracy of the fused ranking.
Because $w$ sees no image or text features, any gain has to come from the validation evidence itself. This makes the key ingredient isolable and keeps the added cost to one BM25 lookup per beam.
The analysis has three parts. We plot $w$ against seen and unseen status and against per-route correctness to check whether the gate learns to distrust the symbolic route on tail entities. We measure how often a failed validation coincides with the dense route being correct. We also test the gate's features one at a time to see which part of the validation evidence carries the signal, since beam agreement and exact match may behave very differently.
Baselines
The naive reference is raw CLIP matching of the image-plus-query embedding against entity embeddings with no gap correction. The file has no reported OVEN accuracy for this CLIP-style dual encoder, so we will measure it ourselves on the identical candidate pool.
The strongest dense competitor is GR-CLIP, meaning our dense route used alone. It is the most relevant rival because it already handles the gap cheaply and well: the paper reports gains of up to 26 NDCG@10 points over raw CLIP on MixBench and about 4 points over VLM2Vec at roughly 75 times less compute (Closing the Modality Gap for Mixed Modality Search). We use the same calibration set size as that paper.
The strongest symbolic competitor is the OVEN recipe: generate a name, then ground it with BM25 (Open-domain Visual Entity Recognition). Its best reported figures are 30.6% seen and 12.4% unseen for PaLI-17B. Our generator will be smaller, so we rerun this recipe with our generator rather than compare to PaLI's numbers.
Next comes the best static fusion. It is the same two calibrated routes combined with a single weight chosen by grid search on validation data, as in GR-CLIP's interpolation. This is the baseline that isolates whether adaptivity matters at all.
Two gate baselines have exactly the same capacity and training data as our gate. The first takes the generator's sequence log-probability and beam entropy as its features. The second takes the fused CLIP features, as in continuous attention-style gating.
An oracle that picks the correct route per example gives the upper bound. Ablations then remove each validation feature in turn and replace the text-side anchor with the raw BM25 score, which tests whether moving to the text side of the gap matters beyond lexical matching.
All systems share the frozen encoders, the generator, the candidate pool, the beam width and the tuning split, and every gate gets the same hyperparameter search budget. Beating the strongest competitor means a statistically significant accuracy gain over both the best static fusion and the confidence gate on the unseen split, using a paired bootstrap over queries, with no loss on the seen split.
Why not just…?
Why not just mean-center CLIP and stop there? GR-CLIP is cheap and strong, and it may already capture most of the achievable gain on seen entities. However, it applies one global translation to every input, and AlignCLIP argues that such shifts blur fine distinctions. Fine distinctions are exactly what separates neighboring Wikipedia entities. The comparison between the dense route alone and the fused system settles this question directly. If fusion adds nothing over GR-CLIP, the proposal fails.
Why not just use a bigger generator? Scale helps, but PaLI-17B still reaches only 12.4% on unseen entities, and tRAG reports that its "seen term bias", where generators reuse familiar terms instead of producing new valid ones, persists in large models. We will run two generator sizes. If the gain from validation-driven fusion shrinks toward zero as the generator grows, we will report the size at which it vanishes.
Why not just tune one fixed fusion weight? This baseline is plausible if both routes fail at similar rates across inputs. Our hypothesis says the routes fail on different inputs, the symbolic route on tail entities and the dense route on visually confusable ones. The oracle-gate gap measures how much per-example routing could possibly gain. If that headroom is small, adaptivity is not worth pursuing.
Why not just gate on the generator's own confidence? This is the real threat. A generator that invents a name may also assign it low probability, in which case validation adds nothing. Hallucinated but fluent names are known to receive high internal probability, which is precisely why external dictionary filters helped so much in the normalization and transliteration work. The matched-capacity comparison between the two gates is the decisive test, and we will also test a gate that combines both feature sets. If log-probability matches validation, the honest conclusion is that the external lexicon is unnecessary for weighting and is useful only for resolving names to entities.
Proposed Experiments
The primary evaluation is OVEN entity accuracy, reported separately for seen and unseen entities and on the human-verified subset. The secondary evaluation is NDCG@10 on the MixBench OVEN and VisualNews subsets, where the symbolic route resolves generated names against the corpus's own entity titles or article entities. This tests whether the effect carries over from recognition to mixed-modality search.
The decisive checks are three. First, on unseen entities, the validation gate must beat both the best static fusion and the confidence gate. Second, the learned weight must drop on inputs where validation fails and rise where the symbolic route is correct, with a clear separation that we will report as a curve rather than a single number. Third, the ablation that replaces the text-side anchor with raw BM25 scores must lose accuracy. That result would show that the benefit comes from crossing the image–text gap through a verified word, not merely from adding a keyword matcher.
References
- Revisiting NMT for Normalization of Early English Letters
- Back-Transliteration of English Loanwords in Japanese
- tRAG: Term-level Retrieval-Augmented Generation for Domain-Adaptive Retrieval
- Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Closing the Modality Gap for Mixed Modality Search
- Mitigate the Gap: Improving Cross-Modal Alignment in CLIP
- Closing the Modality Gap Aligns Group-Wise Semantics
- Ambiguity-aware Multi-level Incongruity Fusion Network for Multi-Modal Sarcasm Detection
- Facilitating Multimodal Classification via Dynamically Learning Modality Gap