Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video

Yulin Zhang, Cheng Shi, Yang Wang, Sibei Yang

Abstract

Traditional Video Large Language Models (V-LLMs) typically operate in a passive, offline manner, processing short video clips only after a user prompt is received. This paradigm fails in real-world egocentric scenarios—such as assisting a user with complex manual tasks—where video is a continuous stream and the AI must anticipate needs. We present VideoLLM-EyeWO (Eyes Wide Open), a model designed for proactive assistance in streaming egocentric video. By addressing the "Triangle of Impossibility"—balancing proactive coherence, just-in-time responsiveness, and synchronized efficiency—EyeWO demonstrates a significant leap in the utility of AI assistants for wearable and real-time applications.

Introduction

Existing V-LLMs are largely reactive. They wait for a user to finish a query before processing a fixed-length video segment. However, a truly helpful AI assistant must operate on a continuous stream, understanding when to intervene without being prompted. This transition to proactive streaming AI introduces three primary challenges:

  1. Proactive Coherence: Maintaining context over long, multi-turn conversations where the visual evidence for a future answer may not have occurred yet.
  2. Just-in-Time Responsiveness: Identifying the "opportune moment" to speak—neither too early (lacking evidence) nor too late (lacking utility).
  3. Synchronized Efficiency: Processing an ever-growing video stream in real-time without the computational lag that typically accompanies long-context modeling.

VideoLLM-EyeWO addresses these challenges through a specialized data generation engine, a multi-stage training strategy, and a novel dynamic compression mechanism.

Methodology

The ESTP-Gen Data Engine

To train a model for proactive behavior, high-quality streaming data is required. The authors developed ESTP-Gen, an automated pipeline that converts raw egocentric narrations from the Ego4D dataset into proactive dialogue. This engine produced 60,000 single-turn and 20,000 multi-turn proactive QA pairs, providing the necessary foundation for the model to learn temporal reasoning and anticipatory response.

Multi-Stage Training Strategy

The model undergoes a three-stage training process to refine its proactive capabilities:

  • Stage 1: Passive Interval Responsiveness. The model learns the general temporal windows in which a response is appropriate.
  • Stage 2: Active Perception. A new action, a_ask_high, is introduced. This allows the model to proactively request high-resolution frames when it detects uncertainty, enabling it to pinpoint exact moments of event occurrence.
  • Stage 3: Proactive Coherence. This stage focuses on multi-turn dialogues, ensuring the model remains consistent and context-aware over extended interactions.

Proactive Dynamic Compression

To maintain real-time synchronization, EyeWO employs a two-level KV cache compression. When the model predicts that a response is imminent, it retains high-resolution tokens. During periods of low activity, it aggressively compresses past content. This approach allows the model to retain critical information while reducing memory overhead to a fraction of traditional methods.

Evaluation and Benchmarking

The researchers introduced ESTP-Bench (Ego STreaming Proactive Benchmark) to evaluate performance across 14 task types, including object localization and intention prediction. These tasks are categorized into Explicit (visual cues), Implicit (reasoning-based), and Contextual (multi-turn) scenarios.

Performance is measured using the ESTP-F1 metric, a novel evaluation tool that penalizes models for responding too early or too late, emphasizing the importance of "just-in-time" assistance.

Key Results and Insights

  • Performance Gains: VideoLLM-EyeWO outperformed the baseline VideoLLM-Online by 19.2% on the ESTP-Bench.
  • Timing and Accuracy: The model demonstrated superior temporal precision compared to offline models like LLaVA-OneVision and Qwen2-VL, proving that specialized proactive training is essential for streaming tasks.
  • Extreme Efficiency: The proactive dynamic compression mechanism reduced KV cache usage to just 0.11% of the baseline. Despite this massive reduction in data retention, the model's performance actually improved, suggesting that selective memory is more effective than exhaustive memory for streaming tasks.
  • Real-Time Capability: The model maintains an Action Per Second (APS) well above the 2 FPS threshold, ensuring it stays synchronized with real-world events on standard hardware (e.g., NVIDIA A40).

Limitations and Future Work

While VideoLLM-EyeWO represents a significant advancement, challenges remain in Contextual Relative Context (TRC) tasks, where models still struggle to maintain perfect coherence over very long, complex interactions. Additionally, while the compression is efficient, the computational spikes associated with high-resolution frame requests need further optimization for deployment on battery-constrained wearable devices like AR glasses.

Conclusion

VideoLLM-EyeWO shifts the paradigm of Video-LLMs from passive observers to proactive participants. By integrating active perception and dynamic memory management, the model achieves a balance of accuracy, timing, and efficiency, paving the way for the next generation of intelligent, real-time AI assistants.

Summary updated

BibTeX
@inproceedings{zhang2025eyes,
  author = {Yulin Zhang and Cheng Shi and Yang Wang and Sibei Yang},
  title = {{Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=2X0Fy0jqPa},
}
Figure