Abstract
Traditional video highlight detection (HD) models typically operate in an offline manner, requiring access to the entire video sequence to identify significant moments. This latency is prohibitive for real-time applications such as search-and-rescue robotics or live surveillance. We present AHA (Autoregressive Highlight Detection), a novel framework designed for strictly online highlight detection. By integrating a multimodal vision-language backbone with a specialized constant-memory mechanism, AHA identifies important events as they unfold. Despite the lack of future context, AHA outperforms state-of-the-art offline models on standard benchmarks, demonstrating that task-conditioned causal reasoning can be more effective than global temporal context.
Introduction
Video highlight detection is the task of identifying the most important or relevant segments within a video stream. While significant progress has been made in offline HD, real-world streaming scenarios present two primary challenges: causality and memory. Most existing models "cheat" by using future information or post-hoc smoothing, which is impossible in live environments. Furthermore, the computational cost of standard Transformer-based models grows linearly with video length, leading to memory exhaustion in long-form streaming.
AHA addresses these issues by treating highlight detection as a real-time, task-conditioned autoregressive problem. It maintains a constant memory footprint regardless of video length and makes decisions based solely on past and current frames, ensuring true causality.
Methodology
Multimodal Architecture
AHA utilizes a multimodal backbone consisting of SigLIP for visual encoding and a Qwen2-7B decoder for language processing. This allows the model to process both the video stream and a natural language task description (e.g., "Find moments where the robot enters a new room").
Multi-Objective Prediction Heads
To provide a nuanced understanding of video content, AHA employs three decoupled prediction heads:
- Relevance: Measures how well a frame aligns with the specific natural language task.
- Informativeness: Determines if a frame provides new information or is redundant compared to previous segments.
- Uncertainty: Estimates the model's internal confidence in its current prediction.
By separating informativeness from relevance, the model can distinguish between a novel but irrelevant scene (e.g., a robot passing an empty hallway) and a relevant event (e.g., locating a target object).
Dynamic SinkCache
To solve the memory explosion problem, AHA introduces Dynamic SinkCache. Standard Key-Value (KV) caches grow with every new frame. Dynamic SinkCache employs a sliding window for recent temporal context but "permanently anchors" the natural language task tokens as "sinks." This ensures that the model never loses sight of its original objective, even after processing thousands of frames, while keeping memory usage constant.
Uncertainty-Aware Scoring and Robustness
The final highlight score is calculated using a piecewise function that penalizes scores when the uncertainty head reports low confidence. This mechanism reduces false positives in ambiguous or visually noisy scenes. Additionally, the model is trained with Video Quality Dropout, simulating real-world sensor degradations like block noise and motion blur to ensure robustness in the field.
Experimental Evaluation
Datasets
The authors introduced the Human Intuition Highlight Dataset (HIHD), containing ~23k videos. This dataset leverages YouTube's "most replayed" engagement metrics as a proxy for human interest, paired with synthetically generated task-driven captions. Evaluation was also conducted on standard benchmarks (TVSum, Mr.HiSum) and the SCOUT dataset, which features 20+ minute egocentric robotics videos.
Performance
AHA achieved new state-of-the-art (SOTA) results on multiple benchmarks:
- TVSum: +5.9% mAP improvement over previous models.
- Mr.HiSum: +8.3% mAP improvement.
Notably, AHA outperformed "offline" models that had access to the full video, suggesting that grounding the detection in a specific language task is a more powerful signal than having access to future frames.
Efficiency
In long-form video testing, the Dynamic SinkCache allowed AHA to process over 127,000 frames. During these tests, the model utilized only 17% of the memory required by a standard linear cache, proving its viability for continuous edge deployment.
Conclusion
AHA represents a significant step forward for real-time video understanding. By combining causal autoregressive modeling with a constant-memory architecture, it enables high-performance highlight detection in streaming environments. Future work will focus on refining the uncertainty head with human-calibrated confidence labels and exploring distilled versions of the 7B-parameter model for deployment on resource-constrained hardware.
Summary updated