Expand Before You Abstain: Timeline-Aware Verification for Lifelog Question Answering

A wearable camera that photographs your day every few seconds produces an archive of hundreds of thousands of images, tagged with time, GPS and sometimes heart rate. People want to ask it questions like "what did I do right after I put away the Christmas tree?" A language model that answers such questions from retrieved photos (retrieval-augmented generation, or RAG: the system looks up evidence and the model writes an answer from it) has two bad options today. It can answer fluently and often wrongly. Or it can run strict fact-checking and refuse almost everything. The best verified multimodal RAG system for first-person images cut hallucinations from 59.62% to 2.88%, but answered "I don't know" 82.69% of the time (CRUISE).

Our idea is that in a lifelog, a failed fact-check usually means the model saw too narrow a slice of time, not that the answer is unknowable. A single frame of a Christmas tree cannot show whether you were putting it away. The thirty frames around it often can. We propose a generator that splits its draft answer into small claims and checks each one. When a claim fails, the system widens the time window around the evidence and checks again. It abstains only when the widened evidence still does not support the claim, or when a calibrated risk threshold says it should. Claims about time, place and order are checked directly against the timestamps and GPS rather than by a vision model. The test is simple: at a matched hallucination rate, the system should answer many more questions than abstention-based verification does.

Core Claim

On real, continuous multimodal lifelogs, most verification failures in RAG answers can be fixed by retrieving neighboring moments in time. Turning "unverified" into "expand and re-check" therefore moves the whole trade-off curve between hallucination and abstention outward, rather than sliding along it as threshold tuning does.

Three ingredients carry this claim. The first is verification-triggered temporal expansion. The second is symbolic checking of claims that the metadata can answer. The third is a calibrated abstention rule: a final cutoff, set on held-out data, that keeps the hallucination rate under a chosen target. The first is the novel ingredient. The other two make the comparison honest and the guarantee meaningful.

What Exists and Where This Sits

Work that motivates the proposal. CRUISE builds a strong verification pipeline for first-person images. It filters noisy evidence, generates two answers (one from retrieved context, one from the model's own knowledge), and breaks answers into sub-claims that must all pass checking. But it treats each image as an isolated query, so when a check fails its only move is to abstain. TimelineQA builds synthetic, text-only lifelogs of up to 78M entries. On single-event questions, generative RAG reaches only 40.3% exact match against 82.6% for a fine-tuned extractive reader. A table-reasoning model falls from 59.0% with perfect retrieval to about 33% with real retrieval. That paper does not diagnose why generation fails. A clinical study (Longitudinal Clinical Summarization) shuffles the timestamps in patient records and finds summaries do not get worse. LLMs treat time series as an unordered "bag of facts," which is exactly the failure an order-sensitive lifelog question exposes. The hallucination survey (Siren's Song) notes that vision-language models hallucinate more than text-only models, which raises the stakes here.

Work that supplies the method. In lifelog retrieval, adding the frames just before and after a match is already known to help. LifeIR goes from 0.1905 to 0.2652 mean average precision (mAP, a ranking-quality score) on NTCIR-18 Lifelog-6 through multi-round chronological expansion with a Qwen2-VL reranker. The same pipeline names the action-versus-presence confusion ("putting away" versus "seeing" a tree). LifeIR, however, ranks images and never generates or verifies an answer. We reuse its expansion, but trigger it from a failed claim during generation rather than running it as a fixed retrieval step. TRAC uses split conformal prediction (a way to set a threshold on held-out examples so that an error rate is bounded with high probability, without assumptions about the model). It applies this to both retriever and generator on Natural Questions, and we borrow its calibration recipe.

Neighbors we differ from. RaDIO also retrieves again mid-generation. Its trigger is a probe on the model's internal activations, and it searches Wikipedia text. Our trigger is an explicit failed claim, and the new query is a time window around existing evidence. SEEM groups dialogue into structured event frames (who, what, when, where, why) and expands from facts back to whole episodes. It is text-only and builds those frames ahead of time with costly LLM calls. We expand on demand and only when verification asks for it. A-MAC shows that per-item LLM calls dominate latency (over 97%). This constrains us: expansion must be bounded and must reuse cheap image embeddings, not re-caption the archive.

Research Program

The system starts from LifeIR-style retrieval. Blurry frames are filtered, CLIP (a model that places images and text in a shared embedding space) retrieves candidates, and the query is rewritten in first person. A vision-language model then drafts an answer and breaks it into atomic claims, each tied to the frames it cites.

Each claim is assigned a type. Metadata claims cover when something happened, where, in what order, and how many times. They are resolved by direct lookup against timestamps, GPS and event boundaries, with no model involved. This addresses the bag-of-facts failure directly, because order is computed rather than read off by a language model. Visual claims cover what was happening. They go to a separate vision-language judge that sees only the cited frames.

When a visual claim fails, the system takes a short window of frames before and after the cited frame. It drops near-duplicate frames by CLIP similarity and re-judges the claim. The window doubles up to a fixed frame budget. A claim that passes is kept and gets a citation to its window. A claim that never passes is either dropped from the answer, if the answer still stands without it, or sends the whole question to abstention.

Each answered question receives a support score: the minimum judge confidence over its claims. A threshold $\lambda$ on this score is set by split conformal risk control on held-out questions. The rule is to pick the smallest $\lambda$ satisfying

$$\frac{n}{n+1}\hat{R}_n(\lambda) + \frac{1}{n+1} \le \alpha,$$

where $\hat{R}_n(\lambda)$ is the fraction of the $n$ calibration questions that are answered and wrong at threshold $\lambda$, and $\alpha$ is the target hallucination rate. This bounds expected hallucination per question at $\alpha$ when calibration and test questions are exchangeable.

Exchangeability is doubtful within one person's timeline. We therefore calibrate on held-out days and report how far observed risk drifts from the target as the gap between calibration days and test days grows.

The analysis side asks why generative lifelog RAG fails. For every hallucination, we label whether the evidence needed was absent from retrieval, present but too narrow in time, present but misread, or ordered wrongly. If the "too narrow" category is small, the core claim is false, and we will say so. We also run the timestamp-shuffle test from the clinical study on our system and on baselines. Order sensitivity should appear only where order is checked symbolically.

Benchmarks and Data

No question-answering benchmark over real multimodal lifelogs exists in the literature we have. We will build one from the NTCIR-16 Lifelog-4 / LSC'21 collection (183,432 images with biometrics and GPS) and the NTCIR-18 Lifelog-6 collection (725,000+ images over 18 months, 26 topics) (NTCIR-16 overview; LifeIR).

The known-item topics come with judged relevant images. Annotators will write several hundred questions whose answers can be checked against those images and their metadata. Questions are stratified into four kinds: single-moment, before/after, action-versus-presence, and counting across days. Answerable and unanswerable versions are balanced so that abstention can be scored.

We will also run on TimelineQA, to test the symbolic-checking and expansion logic at text-only scale against its published numbers. We will run on CRAG-MM, to show the method does not hurt in CRUISE's own setting, where temporal context is absent and expansion should rarely fire.

Baselines

The do-nothing reference is a vision-language model answering from the top CLIP-retrieved frames with no verification. This is the analogue of the baseline RAG agent whose truthfulness score was -34.62 on CRAG-MM in CRUISE. Every other system must beat it on truthfulness.

The strongest existing method is CRUISE's verify-or-abstain pipeline, reimplemented on our retrieval stack. It keeps evidence filtering, two-path generation and sub-claim checking with "I don't know" on any failure. It is the strongest competitor because it is the only published system targeting hallucination in first-person multimodal RAG. Its reported operating point on CRAG-MM is 2.88% hallucination, 82.69% missing and a truthfulness score of 11.54 (CRUISE). On our new lifelog benchmark, no reported result exists on Lacuna for any QA system, so we will run it ourselves.

For fairness, all systems share the same retriever, the same vision-language judge, the same frame budget per question, the same calibration split, and the same hyperparameter search size. We will also sweep CRUISE's abstention threshold to trace its full hallucination-versus-missing curve. Beating CRUISE means a lower missing rate at an equal or lower hallucination rate, across the whole curve rather than at one hand-picked point.

The extractive/structured baseline converts frames to captions and metadata rows. It answers with a fine-tuned extractive reader plus SQL-style aggregation over metadata. This is the lifelog version of TimelineQA's 82.6%-exact-match reader (TimelineQA). It cannot hallucinate free text, so it sets the bar for single-moment questions.

The key ablation is a fixed-window variant. It gives the generator LifeIR's expanded temporal window up front, with the same total frame budget, and then runs CRUISE-style verification. This variant isolates whether expansion triggered by verification failure matters, or whether more temporal context alone does the work. Further ablations remove the symbolic metadata checker (all claims go to the vision model) and remove conformal calibration (a fixed threshold).

Why not just…?

Why not just give the model a wider time window from the start? This is the most serious threat, because LifeIR already shows that wider windows improve retrieval (LifeIR). Our counter-argument is that wide windows given up front add many near-identical routine frames. Long uncurated context distracts models: SEEM reports a 7B model with structured memory beating larger models on raw long context, and the clinical study finds longer context did not help temporal reasoning (Longitudinal Clinical Summarization). Expanding only the claims that failed spends frames where they are needed. The matched-budget fixed-window ablation settles this. If it matches our system, the contribution shrinks to "use temporal context in verification," and we will report it that way.

Why not just tune the abstention threshold, or calibrate it conformally as TRAC does? Calibration picks a point on an existing curve. It does not change what evidence the verifier sees. TRAC needed prediction sets averaging 20.4 answers to reach 0.90 coverage on Natural Questions, which shows that guarantees on weak evidence buy safety by giving up usefulness. We run calibrated CRUISE as a full curve. If its curve overlaps ours, expansion adds nothing.

Why not just skip generation and use extractive QA over captions and metadata? On TimelineQA's single-event questions this clearly wins against naive generation: 82.6% versus 40.3% exact match (TimelineQA). We take that seriously, and our symbolic checker is partly this baseline built into the system. However, a caption pipeline must decide what to describe before any question is asked. Action-versus-presence distinctions are exactly what frame captions miss (LifeIR). Table reasoning also collapsed to about 33% under real retrieval on multi-hop and counting questions. We report results for each question kind separately. If extractive QA wins on before/after and action questions too, generation is not justified for those kinds.

Why not just put the whole lifelog in a long-context model? A 725,000-frame archive is orders of magnitude beyond any context window. Even within a single day, the shuffle result shows that long context does not make a model respect order (Longitudinal Clinical Summarization). We include a one-day long-context baseline on the same-day subset of questions for completeness.

Decisive Checks

The main result is the hallucination-versus-missing curve on the new NTCIR-derived benchmark, compared against swept CRUISE, the fixed-window ablation and extractive QA. We report truthfulness score (accuracy minus hallucination rate, as in CRAG-MM) and per-question-kind breakdowns. The headline claim holds only if the curve dominates both CRUISE and fixed-window verification at matched frame budgets.

Three secondary checks follow. The first is the failure taxonomy: what share of baseline hallucinations and abstentions fall in the "evidence too narrow in time" category. The second is calibration validity: whether observed hallucination stays at or below $\alpha$ when calibrating on held-out days, and how it drifts as test days move further from calibration days. The third is the timestamp-shuffle test, which should degrade our system on order questions and leave the baselines unchanged.

On CRAG-MM, we expect to roughly reproduce CRUISE's operating point, since expansion has little to grab onto there. A large gain on CRAG-MM would actually suggest our gains come from something other than temporal context. We also report latency and the number of judge calls per question, given A-MAC's warning about LLM-call cost.