Questions like "who designed this bridge?" about a photo fail mostly before any reasoning happens. The system must first guess which real-world entity is in the picture. It must then find the sentence in that entity's encyclopedia article that answers the question, and that sentence may say "architect" where the user said "designer." Current pipelines commit to one entity and a few passages at each step. When either guess is wrong, the reader model answers confidently from the wrong text. We propose to keep several entity guesses and several passage guesses alive, each with a graded degree of plausibility in $[0,1]$. These are combined with possibility theory, a branch of fuzzy logic in which evidence is merged by taking minimums along a chain of reasoning and maximums across alternative chains. The system then answers only when the winning answer is necessary, meaning no competing answer is also highly plausible.
The core bet is specific. Retrieval scores over a fine-grained encyclopedia are uncalibrated, and the encyclopedia is full of near-duplicates: sibling species and similar landmarks whose pages differ little. Summing probability-like weights over such candidates double-counts redundant evidence. Taking a maximum does not, because the maximum of a value with itself is unchanged. We therefore predict that max–min aggregation beats both crisp top-1 commitment and probabilistic marginalization. The gains should be largest on questions where the entity is visually ambiguous or where the question's wording shares no content words with the answering passage. Possibility theory also provides an abstention signal, necessity, that is distinct from entropy.
Why this gap, and what already exists
The case that entity linking is the bottleneck is strong. On INFOSEEK, a pipeline of CLIP plus a Fusion-in-Decoder reader reaches 18.2% on the human split, and giving it the true entity raises that to 45.6% (Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?). On Encyclopedic VQA, PaLM reaches 87.0% with oracle retrieval but 48.8% with Google Lens retrieval, and accuracy collapses when the wrong passage is retrieved (Encyclopedic VQA). These papers motivate the work: errors in the first two steps, not in reading, dominate.
Two lines of work act on those steps.
- Better retrieval. EntRAG binds entity tokens to visual features and reranks candidates by a linear mix of entity similarity and context relevance. It reaches 44.5 on InfoSeek and 46.1 on E-VQA. It still cuts candidates at a crisp top-$k$ and passes them forward as if they were all true.
- Better use of retrieved text. ALFAR shifts attention toward retrieved text and blends the reader's internal-memory and context-based output scores. It raises LLaVA-1.5 on E-VQA from 19.14% to 29.57%. It assumes the retrieved text is the right text, and it requires white-box access to attention and output scores.
We reuse both as components or baselines. Our change is to what happens between them: how uncertainty about the entity and about the passage is carried to the answer.
On the vocabulary side, RQUERY shows that query expansions must be checked against the knowledge base's actual wording. It reaches an MRR of 0.53 on mismatch queries, against 0.34 for n-gram models. However, it maps each user term to exactly one schema term. RaFe and InteR rewrite queries using feedback from a reranker or an LLM. They are text-only, and InteR degrades past two iterations because of added noise. We borrow RQUERY's idea of validating against the knowledge base, but replace its one-to-one choice with graded membership across several candidate passages.
On the fuzzy side, the closest work is Fuzzy Multimodal Learning for Trusted Cross-modal Retrieval (FUME). FUME trains possibility and necessity measures into image–text matching and refuses to answer above a fixed uncertainty of 0.5. It evaluates category-level retrieval, not encyclopedic question answering, and has no notion of a reasoning chain from entity to passage to answer. Fuzzy Contrastive Decoding uses Gaussian membership functions and nine Takagi–Sugeno rules to correct token scores during generation. It works closed-book only.
We deliberately do not propose a fuzzy-rule version of ALFAR's score blending. A small learned gate would very likely do as well, and fuzzy vocabulary would add nothing. The fuzzy ingredient we keep is the one with a distinct mathematical property: idempotent max–min aggregation, plus the possibility/necessity pair.
Core Claim
Each candidate answer $a$ receives a plausibility built from three graded memberships:
- $\mu_E(e)$: how plausibly the image and question show entity $e$.
- $\mu_A(p \mid q, e)$: how plausibly passage $p$ from $e$'s article addresses the attribute asked about in question $q$.
- $\mu_R(a \mid p, q)$: how strongly a reader extracts $a$ from $p$.
These combine as
$$\Pi(a) = \max_{(e,p)} \min\big(\mu_E(e),\ \mu_A(p \mid q,e),\ \mu_R(a \mid p,q)\big), \qquad N(a) = 1 - \max_{a' \neq a} \Pi(a').$$
$\Pi(a)$ is the possibility of answer $a$: its best supporting chain is only as strong as that chain's weakest link. $N(a)$ is its necessity. The system outputs $\arg\max_a \Pi(a)$ and abstains when $N$ of that answer falls below a threshold.
Two behaviours follow that crisp and probabilistic pipelines lack.
- Answering through entity ambiguity. When the top entities are near-twins that share the asked attribute, they support the same answer. $N$ stays high, so the system answers even though it cannot tell the entities apart.
- Robustness to duplicate pages. When the knowledge base contains duplicates of a wrong entity, $\Pi$ for the wrong answer does not grow. Sum-based marginalization would grow it.
Research Program
The entity membership $\mu_E$ comes from the same visual–text retriever used by the baselines, the EntRAG-style binding or CLIP. We map its raw score to $[0,1]$ with a monotone (isotonic) function fit on development data. This answers the field's standing complaint that membership functions are hand-set.
The attribute membership $\mu_A$ handles the vocabulary gap. A cross-encoder reranker scores each section of a candidate entity's article against the question, so "designer" can match an "architect" passage even with no shared words. We combine this score with an RQUERY-style check: does an expansion of the question's attribute word appear among the article's literals or the knowledge base's property names? The result is again isotonically calibrated. Because many passages keep nonzero membership, many-to-many matches between question wording and article wording are allowed. RQUERY's one-to-one restriction does not apply.
The reader membership $\mu_R$ comes from the reader's normalized answer score. For API-only models, it comes from answer frequency over samples. This makes the method usable where ALFAR is not.
Two practical points:
- Cost control. The reader runs on each (entity, passage) pair separately. We cap the candidates at $k_E$ entities times $k_P$ passages and prune pairs whose $\min(\mu_E,\mu_A)$ already falls below the current best $\Pi$. Max–min permits this kind of early pruning.
- Abstention threshold. The necessity threshold is chosen on development data to hit a target coverage, unlike FUME's fixed $u=0.5$.
Baselines
The naive reference points are closed-book readers and crisp top-1 pipelines.
- Closed-book readers. On E-VQA, PaLM scores 19.7% and PaLI 13.0% (Encyclopedic VQA). On INFOSEEK-Human, PaLI-X scores 10.8% (Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?).
- Crisp top-1 pipelines. CLIP plus FiD reaches 18.2% on INFOSEEK-Human; PaLM with Google Lens reaches 48.8% on E-VQA.
We rerun the crisp pipeline ourselves with our exact retriever and reader, taking only the top entity and top passages. That gives the "do nothing about uncertainty" point under identical components.
The strongest retrieval-side competitor is EntRAG, reporting 44.5 on InfoSeek and 46.1 on E-VQA. It is the strongest because it already attacks the entity bottleneck directly. No EntRAG code appears in the file. We will therefore reimplement its linear reranking and top-$k$ cutoff on top of our shared retriever. We report both our reimplementation and its published numbers, and flag any mismatch in splits or readers. EntRAG's 46.1 and PaLM+Lens's 48.8 on E-VQA use different readers and setups, so we do not treat either as a clean target.
The strongest knowledge-use competitor is ALFAR. Its reported results are 19.14% to 29.57% on E-VQA with LLaVA-1.5, and a 6.38% gain on InfoSeek multiple-choice over standard retrieval-augmented generation. We run its released code (Lackel/ALFAR) with LLaVA-1.5 on the same retrieved candidates. Because ALFAR improves the reader rather than the aggregation, we also test it as the reader inside our method.
Fairness rules apply to every method:
- the same reader and retriever;
- the same $k_E \times k_P$ candidate pool;
- the same development split for calibration and threshold selection;
- an equal hyperparameter search budget.
Beating ALFAR means higher accuracy on E-VQA with LLaVA-1.5 under these conditions than its reproduced score. Beating our EntRAG reimplementation means higher accuracy at matched candidate count.
The decisive ablation keeps every membership score fixed and changes only the aggregation. We compare max–min against two alternatives:
- sum-product marginalization: a sum over chains of products of softmax-normalized, temperature-calibrated scores;
- a learned aggregator: a small network over the same scores.
This isolates whether the possibilistic algebra, rather than simply keeping multiple candidates, drives any gain.
Why not just…?
Why not just use a bigger closed-book model? The file's evidence is decisive at current scales: 13.0–19.7% on E-VQA and 10.8% on INFOSEEK-Human for large closed-book models. These benchmarks target long-tail entities that are poorly memorized. We include the closed-book reader only as a floor.
Why not just marginalize probabilistically over the top-$k$ entities and passages? This is the real threat. With good calibration it may match max–min on ordinary questions. Our argument depends on redundancy, so we test redundancy directly. We inject near-duplicate copies of distractor entity pages into the knowledge base. Our prediction is that sum-product accuracy falls as copies are added while max–min accuracy stays flat. If calibrated marginalization matches max–min both on the clean benchmark and under duplication, the possibilistic claim fails. We would then report it as a negative result rather than repackage the method.
Why not just expand the question with an LLM and retrieve once? InteR shows LLM expansion helps text retrieval but adds noise beyond two rounds. RQUERY shows unvalidated expansion hurts. We include an InteR-style two-round expansion feeding the crisp pipeline as a baseline. If it closes the gap on the low-overlap subset, graded passage membership is unnecessary for the vocabulary problem. The two ideas are also compatible: expansion can feed $\mu_A$.
Why not just raise $k$ and let the reader see everything? E-VQA reports that wrong passages pull reader accuracy down sharply. Concatenating many candidates hands the reader exactly that noise. Our design reads each chain separately and lets aggregation, not attention, decide. We run a concatenate-all-candidates control at the same $k_E \times k_P$ to measure this.
Proposed Experiments
Benchmarks and metrics. We evaluate on E-VQA, including its two-hop subset, and on INFOSEEK (Wikidata and Human splits, using the 100k-entity knowledge base). We report:
- each benchmark's official accuracy;
- top-1 entity recall;
- risk–coverage curves and their area for abstention.
For abstention we compare necessity against reader max-probability, entropy, and FUME-style fixed thresholding.
Stratified analysis. We split results along two axes:
- entity ambiguity, measured as the score gap between the first and second entity candidates;
- lexical overlap, comparing questions whose gold passage shares no content words with the question against all others.
The claim predicts that gains concentrate in the ambiguous-entity and zero-overlap cells, and are near zero elsewhere.
Decisive checks. Four results would settle the proposal:
- the aggregation ablation under fixed scores;
- the duplicate-injection stress test;
- whether necessity-based abstention yields lower risk at matched coverage than entropy;
- whether the answered-through-ambiguity cases, where $N$ is high despite a small entity score gap, are in fact correct at rates close to the oracle-entity setting.
Failing the first two checks would mean the contribution is "keep multiple candidates," not possibility theory. We would state that plainly.
References
- Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
- Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
- EntRAG: Entity-Centric Retrieval-Augmented Generation for Knowledge-based Visual Question Answering
- Boosting Knowledge Utilization in Multimodal Large Language Models via Adaptive Logits Fusion and Attention Reallocation
- RQUERY: Rewriting Natural Language Queries on Knowledge Graphs to Alleviate the Vocabulary Mismatch Problem
- RaFe: Ranking Feedback Improves Query Rewriting for RAG
- Synergistic Interplay between Search and Large Language Models for Information Retrieval
- Fuzzy Multimodal Learning for Trusted Cross-modal Retrieval
- Fuzzy Contrastive Decoding to Alleviate Object Hallucination in Large Vision-Language Models