Query-Side Scoring Functions for Visual Entity Lookup Over a Growing Knowledge Base

Systems that answer questions about photos using an encyclopedia fail mainly for one reason: they misidentify which entity (which species, which bridge, which building) the photo shows. When the correct entity is handed to them, accuracy more than doubles. The most accurate fix today is a cross-encoder, a model that reads the photo, the question and each candidate entity together in one transformer pass. It is slow because it needs a full pass for every candidate. It also breaks down when the encyclopedia keeps growing, because every update to the model forces every stored entry to be re-encoded, and fine-tuning on new entities erodes accuracy on old ones.

We propose to put all of the model's expressiveness, and all of its learning over time, on the query side. A hypernetwork is a small network that outputs the weights of another network. Here it reads the photo and question and generates a small per-query scoring network. That scoring network then rates entity vectors that were computed once by a frozen encoder and never recomputed. Because the entity side never changes, adding new entities is only an append to the index. Because the scorer is generated from the query, forgetting shows up in one place only, the hypernetwork. We protect it with a function-space penalty whose anchor queries are built from the knowledge base itself, so no user data needs to be stored.

Core Claim

A query-conditioned non-linear scorer over a frozen entity index can recover most of the entity-grounding accuracy of cross-attention reranking, at bi-encoder-like latency. When new entity domains arrive in sequence, it can keep old-domain accuracy without re-indexing and without gradient replay of stored queries. What is new is the combination of three choices. We move the Hypencoder idea into multimodal entity grounding. We freeze the entity side on purpose, so that index consistency holds by construction. And we make the knowledge base double as the rehearsal set. The claim is falsifiable in two ways. It fails if a simple frozen-encoder baseline with appended entities already retains and transfers equally well. It also fails if a bi-encoder plus cross-encoder cascade dominates at matched latency.

Where This Sits

Three papers motivate the gap by showing that entity identification is the bottleneck. Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? introduces INFOSEEK. On its human-written split, a CLIP-plus-FiD pipeline reaches 18.2% while the same reader with oracle entities reaches 45.6%. Encyclopedic VQA reports the same pattern: PaLM gets 48.8% with Google Lens retrieval and 87.0% with oracle retrieval. EntRAG closes part of that gap to 46.1% on E-VQA and 44.5% on InfoSeek. It does so with EntBind, which binds entity text to visual patches, and ECRanker, a cross-attention reranker. The authors state that this makes inference slower, and they assume a fixed entity set. Our proposal keeps EntRAG's insight that grounding must be fine-grained and multimodal. We change where the fine-grained computation happens: inside a generated query function, not a per-candidate joint pass.

Two papers supply the method. Hypencoder generates a small feed-forward "q-net" from contextualized query tokens and scores precomputed 768-dimensional passage vectors with it. It argues via Radon's theorem that inner products cannot express some relevance patterns, reports 0.736 nDCG@10 on TREC DL '19/'20 and 59.6 ms per query over 8.8M passages using graph-based greedy search, and was only tested on a static text corpus. Continual learning with hypernetworks shows that regularizing a hypernetwork's outputs on past task embeddings prevents forgetting, with about 96% accuracy across 100 Permuted MNIST tasks. We borrow that output-regularization idea but apply it per query and in score space, with no task identifiers at test time.

Three papers constrain the design. Replay can provably increase forgetting proves that gradient steps on small replay buffers can turn vanishing forgetting into constant forgetting. That result is why we use a matching penalty rather than replaying old task losses, though we do not assume the theorem is irrelevant to penalties and test buffer-size non-monotonicity directly. AURORA treats index maintenance as few-shot continual learning. It reports up to +26.9% Recall@10 on novel topics and adapts in 28 ms, but only for text dense, sparse and generative indices. We differ by never touching the index representation, and by working with non-linear multimodal scoring. REVEAL shows that knowledge re-inserted into an external memory restores 6.7% accuracy with no weight change. This is the strongest argument for the "freeze everything and append" baseline, which we take seriously below. Overcoming Dual Drift for Continual Long-Tailed VQA handles forgetting in VQA only as closed-vocabulary classification, so it cannot represent unseen entities.

Research Program

Frozen entity side. Each entity in the E-VQA Wikipedia set and the InfoSeek entity set becomes one vector from a frozen image-text encoder. The vector combines the entity's reference image(s) with its title and summary. These vectors are built once and stored in a small-world graph index.

Query side and scorer. The query side reads the photo's patch tokens and the question tokens. An attention-based hyper-head, as in Hypencoder, emits the weights of a 4-to-6-layer q-net. Letting the hyper-head attend to patches is how we aim to capture the localized entity detail that EntRAG found CLIP misses. Retrieval uses Hypencoder's greedy graph search. The top entity's Wikipedia section is passed to a fixed answer generator, so accuracy differences are attributable to grounding.

Retention penalty. When a new domain arrives at stage $t$, we train the hypernetwork parameters $\theta$ on the new domain's questions with a contrastive grounding loss $\mathcal{L}_t$, and we add a function-space penalty:

$$\mathcal{L} = \mathcal{L}_t(\theta) + \lambda \sum_{a \in \mathcal{A}_{<t}} \sum_{e \in \mathcal{N}(a)} \big(s_\theta(a,e) - s_{\theta_{t-1}}(a,e)\big)^2 .$$

Here $s_\theta(a,e)$ is the score that the q-net generated for anchor query $a$ assigns to entity vector $e$. The set $\mathcal{N}(a)$ contains the anchor's true entity and its hardest indexed neighbours, $\theta_{t-1}$ is the previous stage's frozen copy, and $\lambda$ sets the retention strength.

Anchor queries. The anchors $\mathcal{A}_{<t}$ are synthesized from the knowledge base: an old entity's alternate Wikipedia or Commons image paired with a templated question. User queries are never stored. Because old entity vectors are already in the index, the penalty costs only q-net evaluations. This matches predictions the way distillation does and involves no replay of the old task loss.

Analysis. We will also measure how far generated q-net weights for fixed anchors drift across stages. This tests whether forgetting is concentrated in a small subspace of the hypernetwork output.

Continual streams. We build the stream from existing benchmarks. E-VQA is partitioned by iNaturalist super-category and by GLDv2 region, and InfoSeek by entity type, giving roughly eight sequential domains. Each domain registers its entities in the index and supplies its training questions. Test questions cover all domains seen so far, and no domain identifier is available at test time.

Baselines

Freeze-and-append. This is the naive reference point and a serious competitor. The frozen image-text encoder is used as a plain dot-product bi-encoder with no training at all, and new entities are simply appended, following REVEAL's plug-and-play logic. It cannot forget. The proposal must beat it in grounding recall on new domains, or else query-side learning buys nothing.

Sequential fine-tuning without the penalty. The same hypernetwork scorer is fine-tuned on each domain with no retention term. This is the ablation that isolates the retention term, and it shows how much forgetting the scorer suffers by default.

Strongest accuracy method. This is an EntRAG-style cascade: bi-encoder retrieval of top-$M$ entities followed by a cross-attention reranker in the ECRanker style. The reranker is fine-tuned per domain with small-buffer replay, and we run a second variant where it is retrained jointly on all seen domains. The reported static reference points are 46.1% VQA accuracy on E-VQA and 44.5% on InfoSeek (EntRAG). We found no reported latency figure on Lacuna for EntRAG, and no reported continual-stream result for any KB-VQA system. We will therefore measure cascade latency ourselves on the same GPU, sweeping $M$.

Fairness and upper bound. All systems share the entity encoder, the index, the answer generator and the per-stage tuning budget. We also train the scorer jointly on all domains, as an upper bound on the joint-training side. Beating the cascade means one of two outcomes at each stage's end. Either we match its grounding recall within two points at one tenth of its per-query latency, or we exceed it at equal latency (small $M$). Old-domain average forgetting must not exceed that of the replay-trained cascade.

Ablations. Two further ablations isolate the key ingredients. One replaces the q-net with a dot product while keeping the retention penalty. The other replaces knowledge-base-derived anchors with an equal number of stored real queries.

Why not just…?

Why not just freeze everything and append entities? This is the real threat. REVEAL shows that memory updates alone recover knowledge, and a frozen scorer cannot forget. It fails only if new domains need a different notion of query–entity match than the frozen encoder provides. The coarse-scene bias EntRAG reports for CLIP-style matching suggests they do, but that evidence comes from static training, not a stream. The freeze-and-append baseline settles this directly. If it matches our new-domain recall, we drop the retention component and report the negative result.

Why not just use the standard bi-encoder then cross-encoder cascade? It is likely the most accurate option at large $M$. But entities that the first stage misses are lost for good, and the reranker must itself be retrained as domains arrive. We do not claim to beat it at unlimited compute. The latency sweep over $M$ shows whether a q-net over the full graph dominates the cascade's accuracy-latency curve.

Why not just replay a buffer of old queries? Replay avoids the machinery of a penalty. However, Mahdaviyeh and Lucas prove that small buffers can increase forgetting. Our anchor-versus-real-query ablation, run at buffer sizes from tens to thousands per domain, tests whether replay shows the predicted non-monotonic forgetting while the score-matching penalty does not. If replay is monotone and as good, we will say so.

Why not just fine-tune the entity encoder too? That changes every stored vector, which forces re-indexing millions of entries per update. We include it with full re-indexing as an upper-bound variant and report the cost in re-encoded entities alongside the accuracy it gains.

Proposed Experiments and Decisive Checks

Metrics. For each stage we report entity Top-1 and Top-5 recall and downstream VQA accuracy on E-VQA and InfoSeek. We also report average accuracy over all seen domains, average forgetting (peak minus final accuracy per old domain), and backward transfer. Finally, we report per-query latency on a single GPU and the number of entity re-encodings, which is zero by design for our method.

First check (static setting). Does the multimodal q-net close a meaningful fraction of the gap between frozen dot-product grounding and the ECRanker-style reranker? If it recovers less than half, the latency argument does not rescue the method.

Second check (continual setting). Does the retention penalty keep old-domain recall within a few points of joint training? It must do so while still beating freeze-and-append on new domains.

Third check (anchor source). Do knowledge-base-derived anchors retain as well as stored real queries? If they do not, the data-free claim fails, though the architecture can still stand.

Sanity check. We transfer the same continual protocol to BEIR topic splits in text, using NFCorpus as AURORA does. This checks that the retention behaviour is not specific to images.