MemInStream: Memorizing or Merely Recognizing? Rethinking Streaming Video Memory Evaluation

Zhenqi He, Dunyuan Xu, Yang Bo, Kien T. Pham, Yanghao Wang, Shuhan Li, Xintong Guo, Yuyang He, Fang Liu, Huankang Guan, Jingyan Chen, Rui Liu, Long ChenarXiv 2026 · arXiv

TL;DR

MemInStream is a new benchmark designed to rigorously evaluate memory in streaming video models by ensuring questions are both visually solvable and strictly require historical information. By filtering out linguistic shortcuts and recent-frame biases, the authors reveal that many current "memory" models are actually just performing well on recent recognition, struggling significantly with long-term retention and multi-step reasoning.

What Problem Does This Paper Address?

In the world of streaming video understanding, models must process a continuous flow of data, deciding what to keep and what to discard. Evaluating how well they "remember" is surprisingly tricky. The authors identify two major pitfalls in existing benchmarks:

  1. Unverified Memory Solvability: If a model gets a question wrong, is it because it forgot the information, or because it never understood the visual scene in the first place?
  2. Unverified Memory Necessity: If a model gets a question right, did it actually use its memory? Often, models can "cheat" by using linguistic priors (guessing based on the text of the question) or by looking at the most recent frames if the answer is still visible.

Without addressing these, we can't tell if a model is truly memorizing or just recognizing recent patterns.

Figure 1: (a) Memory Solvability. Models may fail to answer questions even when the required evidence intervals are directly provided. Results are reported on the backward tracing tasks of OVO-Bench, the real-time video understanding tasks of StreamingBench, and S-EMBER. (b) Memory Necessity. We compare a memory-based method (FluxMem [1]) with SimpleStream [2], which uses only the four most recent frames, and a text-only baseline. (c) Our evidence-controlled validation protocol ensures both memory solvability and memory necessity, providing a comprehensive benchmark dedicated to evaluating memory capabilities in streaming video understanding.

Methods

The researchers developed an evidence-controlled validation protocol to build the MemInStream benchmark. The core idea is to associate every question with specific "evidence intervals"—the exact timestamps where the answer is visually present.

The protocol applies three strict filters:

  • Solvability Check: A question is only kept if strong models (like Gemini-3.8-Flash) can answer it when only shown the evidence interval. This ensures the failure point isn't basic visual recognition.
  • Text-only Check: If a model can answer the question without seeing any video at all, it's discarded as a linguistic shortcut.
  • Post-Evidence Check: If a model can answer the question using only the frames after the evidence has passed, it's discarded. This ensures the model must have retained information from the past.

The benchmark organizes memory into a hierarchical taxonomy of 17 tasks across four major categories: Episodic Retrieval, State Tracking & Updating, Multi-Evidence Integration, and Contextual Memory.

Datasets

The authors introduced the MemInStream benchmark, which consists of 3,799 high-quality questions and 8,217 annotated evidence intervals.

It was constructed by:

  1. Collecting and re-validating memory-related questions from five existing benchmarks: StreamingBench, OVO-Bench, TeleEgo, PhoStream, and S-EMBER.
  2. Generating new questions using LLMs to cover gaps in real-world scenarios, particularly in state tracking and multi-hop reasoning.
The hierarchical taxonomy and temporal statistics of the MemInStream benchmark.

Key Insights and Ideas

  • The "Recent Frame" Shortcut is Real: Many offline models appear to have "memory" simply because their input window happens to overlap with the evidence. When the evidence is moved further back in time, their performance craters.
  • Integration is Harder than Retrieval: Models are relatively okay at recalling a single fact (Episodic Retrieval), but they struggle immensely when they have to combine information from multiple time points (Multi-Evidence Integration) or maintain a conversation-based context.
  • Memory Augmentation Works: By testing the same model (JoyAI) with and without its memory module enabled, the authors proved that MemInStream successfully measures the utility of memory. The performance gains were most significant in complex tracking and integration tasks.

Key Results

The authors evaluated a wide range of proprietary (Gemini, Doubao) and open-source (Qwen, InternVL, StreamForest) models.

  • Performance Gap: Even top-tier models like Gemini-3.8-Flash saw accuracy drop from ~90% when evidence was in the recent window to ~20% when it was not.
  • Top Performer: Among open-source online models, StreamForest achieved the highest average accuracy (37.1% macro-average), showing particular strength in State Tracking.
  • The "Wall" of Multi-hop Reasoning: Tasks like "Multi-hop Memory Binding" remain a massive challenge, with even the best models scoring only between 13% and 24%.
Accuracy drops significantly as the temporal distance between the evidence and the query increases.

Limitations and Future Work

While MemInStream is a significant step forward, the authors acknowledge that its current scope focuses on videos of moderate length. Future work will need to extend these protocols to prolonged video streams (hours or even days), where the challenges of long-term memory decay, interference from similar events, and efficient retrieval become even more acute. They also suggest that future benchmarks should explore more complex interactive scenarios where the model's past actions influence the video stream it perceives.

Summary updated

Abstract

Streaming video understanding aims to continuously process incoming visual observations, making the retention and retrieval of relevant information increasingly important as the video history grows. Although existing benchmarks include memory-related questions, question-answering accuracy alone does not reliably measure memory capabilities. Incorrect predictions may arise from failures in visual understanding rather than forgetting, whereas correct predictions may rely on linguistic priors or recent observations without requiring historical memory. To address these ambiguities, we introduce MemInStream, a benchmark dedicated to evaluating memory capabilities in streaming video understanding. We develop an evidence-controlled validation protocol that associates each question with temporally localized evidence intervals. The protocol validates evidence solvability and memory necessity by retaining questions that models can answer from the annotated evidence but cannot resolve using recent observations, post-evidence observations, or textual input alone. This helps disentangle memory limitations from understanding failures while reducing memory-independent shortcuts. MemInStream provides a systematic evaluation of memory capabilities through four major categories, covering Episodic Retrieval, State Tracking \& Updating, Multi-Evidence Integration, and Contextual Memory across diverse video scenarios. Overall, \textsc{MemInStream} provides a comprehensive and reliable benchmark for systematically evaluating memory capabilities in streaming video understanding.
BibTeX
@misc{he2026meminstream,
  author = {Zhenqi He and Dunyuan Xu and Yang Bo and Kien T. Pham and Yanghao Wang and Shuhan Li and Xintong Guo and Yuyang He and Fang Liu and Huankang Guan and Jingyan Chen and Rui Liu and Long Chen},
  title = {{MemInStream: Memorizing or Merely Recognizing? Rethinking Streaming Video Memory Evaluation}},
  year = {2026},
  eprint = {2611.15640},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15640},
}
Figure