Entity Ledgers: Keeping Visual Knowledge Retrieval Current Without Retraining

Ask a vision-language model "who designed this bridge?" and it usually fails. The problem is rarely that it cannot see the bridge. It fails because it cannot work out which bridge it is, and so it cannot look up the right facts. Fixing that step alone roughly doubles accuracy. Systems that tie image features to named entities in an encyclopedia now do well on this kind of question, but they are built once, from a fixed snapshot of Wikipedia. The world keeps changing after the snapshot. New landmarks and products appear, places and species get renamed, and facts about them expire. Today the only reliable way to keep such a system current is to re-embed and retrain the entity retriever over millions of entries.

We propose to move all of this change into a small, editable data structure between the image and the document index, called an entity ledger. Each entry holds a handful of visual exemplar embeddings for one entity, its names with the dates each name was in use, and time-stamped facts with the dates they held. A new entity, a renamed entity, or an expired fact becomes an insert or an edit, with no gradient step. The frozen image encoder is touched only when a cheap geometric test says the new entities cannot be told apart in its existing embedding space. The research question is empirical and sharp: how much of real-world knowledge change in visual question answering can be absorbed without training, and can we detect in advance the cases that cannot?

Core Claim

In visual information-seeking, three problems sit at the same place: the entity layer. The first is that retrieval depends on external knowledge. The second is the vocabulary gap between a deictic question ("this bridge") or an outdated name and the terms the documents use. The third is knowledge going stale. Linking the image to an entity ID resolves the vocabulary gap by lookup: every name the entity has ever carried can be used to query documents. The dates on names and facts then resolve staleness by filtering to what was true at the time of the question.

Our claim is that a ledger-plus-router system will match periodic full retraining of the entity retriever at a small fraction of the update cost. It will do so on new-entity accuracy, on retention of old entities, and on questions about renamed entities. It will also clearly beat a frozen system that simply appends new embeddings, but only on the subset of updates its separability test flags. That last clause matters. If the test almost never fires, the honest conclusion is that appending suffices, and we will report that.

Grounding in Prior Work

Three findings motivate the proposal.

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? introduced INFOSEEK and showed that a CLIP-plus-reader pipeline reaches 18.2% accuracy, rising to 45.6% when given the oracle entity. The main failure is therefore linking pixels to an encyclopedia entry. Encyclopedic VQA found the same pattern: PaLM goes from 19.7% alone, to 48.8% with Google Lens, to 87.0% with oracle retrieval. RealTime QA showed that on changing news, 34% of errors come from retrieval failure, after which models fall back on stale memorized answers.

Two works supply methods we reuse.

  • EntRAG binds visual features to entity tokens and reranks by entity similarity. It reaches 44.5 on INFOSEEK and 46.1 on E-VQA, but its index is a static snapshot trained offline. We reuse its entity-centric retrieval as the backbone and replace its static index with the ledger. - Deep Incremental Hashing Network keeps legacy database codes fixed and trains only codes for new categories plus the query-side function. It matches batch accuracy (0.9346 vs 0.9390 MAP on CIFAR-10) at about a third of the training time. We borrow this asymmetry for the case where the encoder must change: legacy ledger embeddings stay frozen and only a query-side adapter is trained.

Several works constrain the design.

  • AURORA is the closest prior work. It treats updating a text retrieval index as few-shot continual learning, adapting 182x faster than retraining and handling vocabulary drift by "lexical surgery." It is text-only and tests drift with synthetic token replacement. Our ledger handles vocabulary change through explicit, dated aliases tied to visually grounded entity IDs, not learned embedding edits.
  • CoMem replays concept-graph triplets to prevent visual forgetting, but it has no way to retire a triplet once it becomes false. The ledger's validity intervals are meant to fill exactly that hole.
  • Mitigating Temporal Misalignment by Discarding Outdated Facts predicts how long a fact typically lasts and triggers retrieval when the time elapsed exceeds that estimate. The authors note it ignores when a state began, and Can Agent Memory Systems Track Evolving State? reinforces that state tracking breaks on sudden changes. We use duration prediction only as a fallback when the ledger has no dated record for a fact.
  • StreamingQA shows that retrieval adapts immediately for rarely mentioned entities, while frequently mentioned ones benefit from parametric updates. That finding is a caution against assuming retrieval alone always suffices, and it is why the router exists.

Research Program

The ledger. Each entity $e$ stores:

  • exemplar embeddings from the frozen image encoder;
  • an alias set $\{(a, [t_0, t_1])\}$ of names $a$ with their periods of use;
  • fact records $(r, v, [t_0, t_1])$ giving a relation $r$ and value $v$ with their validity interval.

Answering a question proceeds in four steps.

  1. Link the image to the top candidate entities by nearest-neighbor search over exemplars, followed by EntRAG-style reranking.
  2. Expand the text query using all aliases of those entities, current and historical. This handles documents written before and after a rename. It uses exact entity-level lookup, unlike RaFe, which trains a free-form query rewriter and inherits the risk of drifting away from the user's intent.
  3. Retrieve documents.
  4. Condition the answer generator on facts whose interval contains the question time $t_q$. Where no dated fact exists, fall back to duration-based discounting.

Updates are inserts: exemplars for new entities, new alias intervals for renames, and closed intervals for expired facts.

The router decides whether an insert is enough. For a batch of new entities with exemplar sets, we compute a leave-one-out margin in the frozen embedding space:

$$\mu = \frac{1}{|X|}\sum_{x \in X}\Big(\max_{x' \in X_{e(x)} \setminus \{x\}} s(x,x') - \max_{x'' \notin X_{e(x)}} s(x,x'')\Big)$$

Here $X$ is the set of new exemplars, $X_{e(x)}$ is the set of exemplars of the same entity as $x$, and $s$ is cosine similarity. A negative $\mu$ means new exemplars sit closer to other entities than to their own.

If $\mu$ is above a threshold $\tau$, which is calibrated once on a held-out slice, the update is insert-only. Otherwise we train a small query-side adapter against frozen legacy embeddings, following the DIHN recipe. To avoid rewriting the old index, we also apply a distillation penalty on old-entity queries in the spirit of Learning without Forgetting.

The stream. No benchmark in our file streams multimodal entity knowledge over time, so we build one from existing ones.

  • Entity growth. Hold out a random 30% of entities in INFOSEEK and E-VQA (whose E-VQA portion spans iNat21 species and GLDv2 landmarks) and release them in five batches.
  • Renaming. Collect the subset of entities with documented name changes in the underlying knowledge base, and add a matched synthetic-rename control so our results can be compared with AURORA's protocol.
  • Fact change. Attach dated answers where the knowledge source records them, and use SituatedQA for text-only temporal questions to validate the interval-filtering component.

Natural renames may be few. We will report their count and treat the synthetic control as secondary.

Baselines

Do nothing. A static EntRAG-style system built on the initial snapshot and never updated. It sets the floor for new-entity accuracy and the staleness rate on changed facts. For reference, EntRAG reports 44.5 VQA accuracy on INFOSEEK and 46.1 on E-VQA (EntRAG), though on full static splits rather than our held-out stream.

Frozen append. The strongest competitor on the core question. It is the same frozen encoder and reranker; new entities' image embeddings and documents are appended to the vector index, and new names are appended as plain text with no intervals. It shares our cost profile, so any gain must come from the router and the dated aliases. It runs on identical data, identical retrieval depth, and the same reader. Beating it means a statistically reliable gain on router-flagged batches and on renamed-entity and changed-fact questions, with no loss elsewhere.

Periodic full retraining. Retrain the entity retriever on all data after each batch. This is the accuracy ceiling and the cost reference. We give it a generous tuning budget and report GPU-hours. We count it as matched if we come within two accuracy points while using under a tenth of its update compute. The 10x cost figure is a target we set, not a result from the file.

Always-adapter. An ablation that trains the adapter on every batch, with the LwF-style penalty. It tells us whether the router's selectivity is worth anything.

AURORA-style text adaptation. Its gated retriever update is reimplemented on the text side of the pipeline. Only its reported text-domain numbers are available: 182x faster adaptation and up to +26.9% Recall@10 on novel topics (AURORA). There is no reported result on Lacuna for AURORA on INFOSEEK or E-VQA.

Ledger ablations. The ledger with intervals removed (all aliases and facts treated as current) isolates the temporal ingredient. The ledger with alias expansion removed (current name only) isolates the vocabulary ingredient.

Ceilings. The oracle-entity ceilings frame the gap: 45.6% on INFOSEEK with FiD, and 87.0% for PaLM on E-VQA.

Why not just…?

Why not just append new embeddings to a frozen index? This is the real threat, and the proposal is built around it: appending is the default path. What frozen appending cannot do is tell whether new fine-grained entities are separable. Species and landmark look-alikes are exactly where CLIP-style embeddings capture the scene rather than the entity, which EntRAG reports as its motivation. Frozen appending also cannot express that a name or fact has stopped being true. If frozen appending matches the full system on flagged batches, the router adds nothing, and we will say so.

Why not just use a commercial image search plus a multimodal LLM? On E-VQA, PaLM with Google Lens reaches 48.8% against 87.0% with oracle retrieval (Encyclopedic VQA). With Google search on live news, 34% of errors remain retrieval failures (RealTime QA). Web search also provides no controlled notion of when a fact held. We will still run a search-API pipeline on a subsample as a reference point.

Why not just retrain periodically? StreamingQA found that incremental fine-tuning used 95% fewer steps while roughly matching retraining, so retraining is not necessary to stay accurate. The question is whether even incremental fine-tuning is needed for entity-level change. Retraining is our ceiling, so the experiment answers this directly.

Why not just predict fact durations instead of storing intervals? Duration prediction cut calibration error by 50–60% on SituatedQA and reached 20.5 EM with adaptive retrieval (Mitigating Temporal Misalignment). However, it predicts an average lifespan and ignores when a state began, so it misses abrupt changes. We keep it as the fallback and compare interval filtering against it on changed-fact questions.

Proposed Experiments and Decisive Checks

Metrics. - Accuracy on new entities after each batch. - Retention: accuracy on the original entities and average forgetting. - Accuracy on renamed-entity questions, split by whether the supporting document uses the old or new name. - Stale-answer rate on changed facts. - Update compute per batch and query latency.

Check 1: router validity. Across batches, the margin $\mu$ should predict the accuracy gap between frozen append and always-adapter, with a rank correlation reported per dataset. If it does not, the router is unjustified.

Check 2: vocabulary gap. On renamed entities whose documents use only the old name, alias expansion should recover most of the accuracy that the current-name-only ablation loses. WikiQA found that 20.3% of correct answers share no content words with the question, which indicates how large pure-lexical failures can be.

Check 3: staleness. Interval filtering should cut the stale-answer rate below both the interval-free ablation and duration-only discounting, without lowering accuracy on facts that have not changed.

Check 4: cost. The full system should stay within two points of periodic retraining on new-entity accuracy while using a small fraction of its GPU-hours.

Failure outcome. If the router rarely fires and frozen append ties it, the publishable finding is that entity-level knowledge change in visual question answering needs no training at all. That is a useful negative result, and the dated-alias results would stand on their own.