When the Index Forgets: Maintenance-Induced Hallucination in Streaming Video Memory

Video assistants that run for hours or days, such as egocentric helpers, monitoring systems, or edge devices, cannot keep every frame in the model's context. They embed frames into vectors and store them in a nearest-neighbor index, which is a data structure that returns stored vectors similar to a query without scanning everything. When the user asks a question, the system retrieves frames from that index and hands them to a vision-language model. The index is updated continuously and is never fully rebuilt. Indexes maintained this way do not miss neighbors at random. Their errors concentrate on recently inserted, unusual content, because new vectors sit in partitions built for older data. Questions about what changed, what happened next, or whether the scene moved depend on exactly that content. When the evidence is silently missing, the model answers anyway from language priors. We call this maintenance-induced hallucination. We propose to measure it, and to show that a maintenance policy built to reduce it lowers hallucination at the same latency as current streaming-index policies.

The research has two parts. The first is a measurement design that isolates how much hallucination the index alone causes, by comparing each approximate index against exact search with the same model, the same stream, and the same questions. The second is a method that redirects the existing signals of incremental index maintenance. These signals are partition imbalance, reconstruction error, and how often a partition is queried. In the method they track the risk that evidence has been missed rather than query traffic, and they decide per query whether to search wider or to tell the model that its evidence may be incomplete.

Core Claim

Streaming vector indexes are tuned and reported on average recall@k, which is the fraction of true nearest neighbors returned. Two systems with the same average recall can differ sharply in hallucination, because what matters is which neighbors are missed. Under drift, misses cluster on fresh, out-of-distribution frames, and those are the frames that temporal-sequence and scene-transition questions require. A maintenance policy that ranks partitions by the novelty of what was written to them and by their reconstruction error should close most of the hallucination gap to exact search at matched query throughput. It should do better than a policy driven by query traffic, as in Incremental IVF Index Maintenance for Streaming Vector Search, and better than spending the same latency on a uniformly wider search.

Positioning

The closest system is Venus. It clusters streaming video frames on an edge device, indexes cluster centroids in a hierarchical memory, and selects keyframes adaptively. Venus found the "redundancy trap": retrieving more frames does not improve accuracy monotonically and can cause hallucination. Venus motivates this proposal, but it uses ordinary lightweight clustering. It never studies how index maintenance policy changes what is retrieved, and it treats the model as a black box. RoRA-VLM shows that retrieval fixes knowledge hallucinations that parametric methods cannot, and that noisy retrieved evidence causes new ones. However, it assumes a static image-text corpus. We take both systems' premise that retrieval quality drives grounding, and we move the question to the index itself as it changes.

The method comes from streaming index work. Ada-IVF (Incremental IVF Index Maintenance for Streaming Vector Search) maintains an inverted-file index. That kind of index splits the vector space into cells around centroids and searches only the few cells nearest the query, a number called nprobe. Ada-IVF splits, merges, and locally re-clusters cells when imbalance or reconstruction error crosses a threshold, and it prioritizes cells by "read temperature," meaning how often they are queried. CrackIVF (Cracking Vector Search Indexes) builds its partitions lazily around incoming queries and leaves new insertions in a buffer until queries touch them. Both were evaluated only on recall and throughput over unimodal benchmarks such as SIFT, GloVe, and MSTuring. Both also contain the bias we expect to cause hallucination: maintenance follows past queries, so newly arrived scenes nobody has asked about yet stay poorly indexed. The file's own open problem, index drift under abrupt workload shifts, describes this failure mode. We supply the downstream consequence that makes it matter.

The hallucination literature sets the limits on our design. VidHalluc shows that spatial saliency reweighting (DINO-HEAL) fails on scene-transition hallucinations, which need global context across time. Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering injects precomputed offset vectors chosen by a binary motion label. Head-level and activation interventions such as Vision-aware Head Divergence and SHARP strengthen attention to visual tokens that are present, and Why LVLMs Are More Prone to Hallucinations in Longer Responses documents how attention drifts away from them. None of these methods can recover frames the retriever never returned. We treat them as complementary layers, not competitors, and we test that claim directly.

Research Program

The first component is a streaming benchmark built from existing hallucination sets. We interleave videos from VidHalluc (5,002 videos, 9,295 question-answer pairs across action, temporal-sequence, and scene-transition questions), EventHallusion, and EgoSchema into one long stream sampled at 1–2 frames per second. We arrange the stream in domain blocks so the embedding distribution shifts abruptly, and we add distractor footage from VideoUFO to reach millions of frame vectors. Frames are embedded with a CLIP-style encoder and inserted in stream order.

Each benchmark question is posed at a controlled delay after its source video is inserted, either immediately or after many more insertions. The query combines an anchor frame (the user "pointing at a moment") with the question text. This follows RoRA-VLM's use of an image as the retrieval anchor, and it prevents a metadata lookup from making retrieval trivial. Retrieved frames, a fixed budget of $k$ per question, go to Video-LLaVA, VideoLLaMA2, and Qwen2.5-VL, the models used in the temporal activation-engineering paper.

The second component is the central measurement. For every index policy, we define the index-induced hallucination gap as the drop in benchmark accuracy relative to exact (flat) search, with the model and frame budget held fixed. We break down recall@k and this gap by the age of the evidence since insertion, by its novelty (distance to the nearest existing centroid at insertion time), and by question axis. The hypothesis is falsifiable. If misses are spread evenly across age and novelty, or if the gap tracks average recall alone, the core claim fails and the paper becomes a negative result about the intersection.

The third component is the method, novelty-weighted maintenance with a per-query risk gate. We keep Ada-IVF's split, merge, and local re-clustering operations, but replace read temperature with a write-risk score per partition. Write risk rises with the fraction of recent insertions that were far from their assigned centroid and with the partition's mean reconstruction error, and it decays as the partition is re-clustered. At query time, the probed partitions give a cheap risk estimate:

$$r(q) = \sum_{p \in P(q)} w_p(q)\, \rho_p,$$

where $P(q)$ is the set of partitions probed for query $q$, $w_p(q)$ is the softmax-normalized similarity of $q$ to partition $p$'s centroid, and $\rho_p$ is that partition's current write-risk score. When $r(q)$ exceeds a threshold, the system first widens the search for this query only, by increasing nprobe and scanning the unpartitioned insertion buffer. If risk is still high, it adds one sentence to the prompt saying that recent evidence may be incomplete, so the model can abstain. All thresholds are tuned on a held-out stream segment.

Baselines

The naive reference point is the static index: an IVF index trained once on the first stream block and then only appended to. It shows how bad drift gets with no maintenance at all. The upper reference is exact flat search over every inserted vector. It defines a zero index-induced gap and also sets the latency cost that approximate indexes exist to avoid.

The strongest existing maintenance method is Ada-IVF with its published read-temperature policy. It is the strongest competitor because it already does local, indicator-driven repair, and it reports 85% of a full rebuild's search throughput with 10x higher update throughput on BigANN-SS, plus update time at 62% of LIRE's with 9% higher queries per second on MSTuring10m (Incremental IVF Index Maintenance for Streaming Vector Search). No reported result was found on Lacuna for any streaming index, Ada-IVF included, on a multimodal hallucination metric, so we will produce the first such numbers.

We also run three more systems. CrackIVF represents query-driven construction. Periodic full rebuild uses the rebuild interval that matches Ada-IVF's total compute. Venus-style adaptive keyframe retrieval over a clustered memory represents the closest end-to-end system, with Venus reporting 1.6x to 3.3x fewer transmitted keyframes on Video-MME at comparable accuracy.

Fairness rules apply to every system. All indexes share the same embeddings, the same number of partitions at initialization, the same frame budget $k$, and the same model and decoding settings. Each policy gets the same hyperparameter search budget on the held-out segment, and comparisons are made at matched per-query latency and matched maintenance CPU time. Beating Ada-IVF means a statistically significant reduction in the index-induced gap on VidHalluc temporal-sequence and scene-transition questions at equal or better queries per second and update throughput, with no loss on action questions.

Two ablations isolate the key ingredient. The first keeps Ada-IVF's read-temperature maintenance but adds the query-time risk gate. The second swaps in write-risk maintenance but removes the gate. A third control widens nprobe using only the retrieval score margin, with no index statistics. If this control matches our gate, the index-state signal adds nothing.

Why not just…?

Why not just use exact search over the stream? At the scale of one video, exact search is the right answer, and we expect it to win. Every result is therefore reported as a curve over stream size that marks the crossover where exact search exceeds an interactive latency budget. Cracking Vector Search Indexes formalizes that flat-scan latency grows linearly with collection size, and Venus reports ingestion backlogs on Jetson-class hardware. If the crossover lands beyond realistic deployment sizes, the proposal applies only to long-horizon or multi-camera memories, and we will say so.

Why not just rebuild the index periodically? A rebuild restores recall, but frames inserted since the last rebuild remain poorly indexed. Our delay-controlled questions measure exactly this window. Ada-IVF reports a 10x update-throughput advantage over rebuilding on BigANN-SS. At matched compute, we expect rebuilds to show a sawtooth pattern in hallucination on fresh evidence, and the experiment will show whether that holds.

Why not just search wider for every query? This is the most serious threat. If spending the extra latency uniformly on a larger nprobe or an HNSW graph closes the gap as well as targeted widening does, then the maintenance policy and the risk gate are unnecessary. We test this head on with matched-latency comparisons and with the score-margin-only control described above.

Why not just fix it in decoding? Head reinforcement, as in Vision-aware Head Divergence, and temporal offset vectors improve use of evidence that is present. VidHalluc already shows saliency reweighting failing on scene transitions. We run VHR on top of every index policy. If decoding fixes close the index-induced gap on their own, our claim fails. We expect instead that the gains add up, since a missing frame cannot be attended to.

Proposed Experiments

The decisive check is the stratified gap analysis. Under Ada-IVF and CrackIVF, recall@k and accuracy should fall more steeply for high-novelty, recently inserted evidence than average recall predicts. The effect should be largest on VidHalluc scene-transition and temporal-sequence questions and on EventHallusion misleading-question subsets, and smallest on action questions answerable from a single frame.

The second check compares write-risk maintenance with the risk gate against all baselines at matched latency. We report per-axis accuracy, the "yes" rate on binary questions as a bias indicator (the measure POPE showed can approach 99%), abstention precision (how often abstentions coincide with truly missing evidence), queries per second, and update throughput.

The third check asks whether $r(q)$ predicts hallucination at all. We compute its correlation with per-question error, controlling for retrieval score margin.

The fourth check repeats the main comparison with data-oblivious compression from TurboQuant applied to stored vectors. Quantization error interacts with novelty, and TurboQuant needs no codebook retraining under drift.

Success means three things together: a demonstrated age- and novelty-dependent gap, a measurable reduction of that gap by the proposed policy over matched-latency uniform widening, and gains that add on top of decoding-time interventions.