EgoJev: Finding Evidence in Egocentric Video by Reading VLM Decisions

Chunggi Lee, Hanspeter PfisterarXiv 2026 · arXiv

TL;DR

EgoJev is a training-free frame selection method that enables Vision-Language Models (VLMs) to find brief evidence moments in long egocentric videos by "reading" their internal decision probabilities over labeled thumbnails. By bypassing text generation in favor of raw output distributions, EgoJev significantly improves question-answering accuracy and evidence localization on wearable devices without the need for a second "scorer" model.

What Problem Does This Paper Address?

Smart glasses and wearable cameras record hours of "first-person" (egocentric) video. When a user asks a question like "Where did I put my keys?", the answer usually lies in a tiny, few-second window of that footage.

Current VLMs face a "bottleneck" problem: they can only process a handful of frames at once due to memory and compute constraints. If you sample frames uniformly, you’ll likely miss the critical moment. Existing solutions often rely on a second, smaller model (like CLIP or BLIP) to "score" every frame for relevance, which adds significant overhead. Alternatively, some ask the VLM to write out a timestamp, but LLMs are notoriously poor at precise temporal math and text-based grounding.

The authors of EgoJev ask: Can a single, frozen VLM find the evidence itself, more accurately and efficiently, by looking at its own internal choices?

Methods

EgoJev introduces a "decision readout" approach that treats frame selection as a multiple-choice problem for the VLM.

  1. Labeled Thumbnails: The video is split into segments, and the middle frame of each is shown to the VLM as a small thumbnail with a label (e.g., "Segment 17:").
  2. Reading the Choice: Instead of letting the VLM generate text, EgoJev looks at the output probability distribution for the next token. If the model is more likely to predict the token "17" than "1", it "points" more strongly to that segment. This allows the model to compare all candidates in a single forward pass.
  3. Coarse-to-Fine Refinement: The process happens in two stages. First, a coarse pass over the whole video identifies promising windows. Then, the VLM "looks again" at higher-resolution thumbnails within those top windows to refine its search.
  4. Quantile Placement: Once a final probability distribution ($P^*$) is calculated across the video, EgoJev places its limited "answer frames" at the quantiles of this distribution. This naturally clusters frames where the model is confident and spreads them out where it is uncertain.
  5. Evidence Time ($t^*$): After answering the question, the VLM is asked which frame was most useful. Reading this distribution provides a precise timestamp for the user.
Overview of the EgoJev pipeline: (1) Ask with thumbnails, (2) Read probabilities, (3) Refine top windows, (4) Place frames at quantiles, (5) Answer and return evidence time.

Datasets

The authors evaluated EgoJev on three major egocentric benchmarks:

  • QAEgo4D: Questions about objects and actions in ~8-minute videos from the Ego4D dataset.
  • MyEgo: A benchmark focused on "long-term" memory, asking about moments that happened up to 11 minutes prior.
  • EgoSchema: A diagnostic benchmark for very long-form (3-minute) video understanding.

Key Insights and Ideas

  • Choosing > Writing: VLMs are much better at selecting a label from a provided list than they are at generating a timestamp from scratch. Reading the raw logits (probabilities) captures the model's nuanced "hunch" better than a single hard-coded text output.
  • The Power of "Looking Again": Brief evidence often falls between coarse thumbnails. The coarse-to-fine strategy (refining $k$ windows) is essential for catching these "blink-and-you-miss-it" moments.
  • Quantiles as Optimal Sampling: Placing frames at quantiles is mathematically shown to minimize the 1-Wasserstein distance between the model's evidence distribution and the selected frames. This is a more robust sampling strategy than simply taking the "Top-K" most probable frames.
  • Training-Free Efficiency: Because it uses a frozen VLM and no second model, EgoJev is highly portable. It can even run on an iPhone 15 Pro, making it a viable candidate for real-world wearable assistants.

Key Results

Using the Qwen3.5-4B model with only 8 answer frames, EgoJev achieved:

  • Accuracy Boost: Raised QAEgo4D accuracy from 51.2% (uniform sampling) to 62.0%.
  • Superior Localization: It placed an answer frame within 8 seconds of the actual evidence for 67.2% of questions, significantly outperforming baselines like AKS (53.8%) and BOLT (56.2%).
  • Speed: It is 3.1x faster per question than A.I.R., a recent iterative frame selector that requires a second model.
  • On-Device Performance: On a Mac (M-series), the entire pipeline from recording to answering takes about 18.8 seconds for an 8-minute video.
Figure 3. Accuracy vs. time per question (QAEgo4D, Qwen3.5- 4B, 8 answer frames, one A100 MIG slice). Baselines at the set- tings of Tab. 1, and A.I.R. also at 1/8 fps (59.4% in 33.5 s). The EgoJev line refines k = 0 to 4 windows. Time: mean end-to-end time over 50 questions.

Limitations and Future Work

  • Thumbnail Recognition: EgoJev is limited by what the VLM can perceive in a small thumbnail. If the evidence is too small or blurry to be recognized at low resolution, the "coarse" pass will fail.
  • API Access: The method requires access to the full probability distribution of output tokens. Many popular closed-source APIs (like GPT-4o or Gemini) currently restrict this data, limiting EgoJev primarily to open-weight models.
  • Future Directions: The authors suggest that future work could involve more sophisticated "look again" strategies or integrating EgoJev with even smaller, more specialized models for ultra-low-power wearable use.

Summary updated

Abstract

Questions about long egocentric video often depend on one brief moment, such as the second a hand puts an object down. A vision-language model (VLM) that sees a few evenly spaced frames usually misses it. Training-free methods rank frames with a separate image-text model or ask the VLM to write a time. In a controlled test, the VLM itself locates the moment better when it chooses among labeled frames, and we read this choice from its output distribution. With the same candidates, this choice lands on the evidence more often than image-text scores or a written time span, and it compares all candidates in one decision. Based on this finding, we propose EgoJev, a training-free frame selector for wearable assistants. EgoJev reads where the evidence is from labeled thumbnails and looks again at the windows it points to. It places its few answer frames at the quantiles of the resulting distribution. One frozen VLM selects, answers and returns the evidence time, with no second model. A wearable device can encode its coarse thumbnails while it records. With Qwen3.5-4B and 8 answer frames, EgoJev raises QAEgo4D accuracy from 51.2% with uniform sampling to 62.0%, and it outperforms recent training-free frame selectors on QAEgo4D and MyEgo. A single-level version runs on a Mac and an iPhone.
BibTeX
@misc{lee2026egojev,
  author = {Chunggi Lee and Hanspeter Pfister},
  title = {{EgoJev: Finding Evidence in Egocentric Video by Reading VLM Decisions}},
  year = {2026},
  eprint = {2611.24616},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24616},
}
Figure