TL;DR
MemInStream is a new benchmark designed to rigorously evaluate memory in streaming video models by ensuring questions are both visually solvable and strictly require historical information. By filtering out linguistic shortcuts and recent-frame biases, the authors reveal that many current "memory" models are actually just performing well on recent recognition, struggling significantly with long-term retention and multi-step reasoning.
What Problem Does This Paper Address?
In the world of streaming video understanding, models must process a continuous flow of data, deciding what to keep and what to discard. Evaluating how well they "remember" is surprisingly tricky. The authors identify two major pitfalls in existing benchmarks:
- Unverified Memory Solvability: If a model gets a question wrong, is it because it forgot the information, or because it never understood the visual scene in the first place?
- Unverified Memory Necessity: If a model gets a question right, did it actually use its memory? Often, models can "cheat" by using linguistic priors (guessing based on the text of the question) or by looking at the most recent frames if the answer is still visible.
Without addressing these, we can't tell if a model is truly memorizing or just recognizing recent patterns.
Methods
The researchers developed an evidence-controlled validation protocol to build the MemInStream benchmark. The core idea is to associate every question with specific "evidence intervals"—the exact timestamps where the answer is visually present.
The protocol applies three strict filters:
- Solvability Check: A question is only kept if strong models (like Gemini-3.8-Flash) can answer it when only shown the evidence interval. This ensures the failure point isn't basic visual recognition.
- Text-only Check: If a model can answer the question without seeing any video at all, it's discarded as a linguistic shortcut.
- Post-Evidence Check: If a model can answer the question using only the frames after the evidence has passed, it's discarded. This ensures the model must have retained information from the past.
The benchmark organizes memory into a hierarchical taxonomy of 17 tasks across four major categories: Episodic Retrieval, State Tracking & Updating, Multi-Evidence Integration, and Contextual Memory.
Datasets
The authors introduced the MemInStream benchmark, which consists of 3,799 high-quality questions and 8,217 annotated evidence intervals.
It was constructed by:
- Collecting and re-validating memory-related questions from five existing benchmarks: StreamingBench, OVO-Bench, TeleEgo, PhoStream, and S-EMBER.
- Generating new questions using LLMs to cover gaps in real-world scenarios, particularly in state tracking and multi-hop reasoning.
Key Insights and Ideas
- The "Recent Frame" Shortcut is Real: Many offline models appear to have "memory" simply because their input window happens to overlap with the evidence. When the evidence is moved further back in time, their performance craters.
- Integration is Harder than Retrieval: Models are relatively okay at recalling a single fact (Episodic Retrieval), but they struggle immensely when they have to combine information from multiple time points (Multi-Evidence Integration) or maintain a conversation-based context.
- Memory Augmentation Works: By testing the same model (JoyAI) with and without its memory module enabled, the authors proved that MemInStream successfully measures the utility of memory. The performance gains were most significant in complex tracking and integration tasks.
Key Results
The authors evaluated a wide range of proprietary (Gemini, Doubao) and open-source (Qwen, InternVL, StreamForest) models.
- Performance Gap: Even top-tier models like Gemini-3.8-Flash saw accuracy drop from ~90% when evidence was in the recent window to ~20% when it was not.
- Top Performer: Among open-source online models, StreamForest achieved the highest average accuracy (37.1% macro-average), showing particular strength in State Tracking.
- The "Wall" of Multi-hop Reasoning: Tasks like "Multi-hop Memory Binding" remain a massive challenge, with even the best models scoring only between 13% and 24%.
Limitations and Future Work
While MemInStream is a significant step forward, the authors acknowledge that its current scope focuses on videos of moderate length. Future work will need to extend these protocols to prolonged video streams (hours or even days), where the challenges of long-term memory decay, interference from similar events, and efficient retrieval become even more acute. They also suggest that future benchmarks should explore more complex interactive scenarios where the model's past actions influence the video stream it perceives.
Summary updated