Abstract
Traditional video recognition models for long-untrimmed content typically rely on a two-stage "preview-then-recognize" paradigm. While effective at reducing total frame processing, this approach introduces computational redundancy and potential information loss between stages. This paper introduces ViMo (View while Moving), a novel hierarchical framework that unifies sampling and recognition into a single-pass process. By utilizing adaptive local unit locators and a global integration module, ViMo efficiently navigates long-form video content to achieve state-of-the-art accuracy with significantly reduced latency.
Introduction
Analyzing long-untrimmed videos—such as a multi-minute recording of a sporting event or a procedural task—presents a significant challenge for computer vision. These videos are often composed of distinct "semantic units" interspersed with irrelevant or redundant frames. Current state-of-the-art efficient models attempt to solve this by first scanning the video with a lightweight model to identify salient frames, then revisiting those frames with a heavier recognition model.
However, this two-stage approach suffers from two primary drawbacks:
- Redundancy: Accessing raw frames twice is computationally expensive and difficult to parallelize.
- Information Loss: The features extracted during the preview stage are often discarded, even though they contain valuable context.
ViMo addresses these issues by treating video recognition as an active exploration task, mimicking the way humans browse videos by skipping through the progress bar while simultaneously building a narrative understanding.
The ViMo Framework
ViMo replaces the linear pipeline with a hierarchical, adaptive system that processes video in a single pass. The architecture is divided into two primary components:
1. Local Unit Locators
To handle the complexity of long videos, ViMo divides the content into local segments. Each segment is assigned a "locator"—an agent consisting of a Spatial Network (vision), a Temporal Network (memory), and a Policy Network (decision-making). Using Reinforcement Learning (specifically Soft Actor-Critic), these locators adaptively decide which frames to "view" and when to stop "moving" within their specific segment. This allows the model to ignore noisy or irrelevant frames dynamically.
2. Multi-unit Integration Module
Once the local scouts have extracted features from their respective segments, a global integration module—typically a Transformer—aggregates these findings. This module is responsible for reasoning about long-range dependencies between the different semantic units, ensuring that the model understands the video as a cohesive story rather than a collection of isolated clips.
By accessing raw frames only once during inference, ViMo achieves a unified process that combines sampling and recognition, maximizing both speed and feature retention.
Experimental Results
ViMo was evaluated across three major benchmarks, demonstrating a superior accuracy-efficiency trade-off compared to existing adaptive methods.
Performance on Long Videos
- ActivityNet-v1.3: ViMo achieved 82.4% mAP using only 38.7 GFLOPs. This outperforms previous leaders like AdaFocus v2 (79.0% mAP), providing a significantly higher accuracy ceiling for a manageable increase in computational cost.
- FCVID: On this dataset of very long social and procedural videos, ViMo reached 86.4% mAP, setting a new benchmark for adaptive recognition.
Efficiency and Versatility
In practical hardware tests on an NVIDIA RTX 3090, ViMo demonstrated a throughput of 129.5 videos per second, significantly outperforming traditional two-stage paradigms in both latency and volume. Furthermore, the model proved versatile; even on the short-trimmed Kinetics-Sounds dataset, it achieved 92.4% accuracy, proving that the "view while moving" strategy is effective regardless of video length.
Discussion and Future Work
ViMo demonstrates that hierarchical reasoning and adaptive sampling are key to efficient video understanding. By treating the video as a series of semantic units rather than a monolithic block, the model effectively filters noise and focuses on high-value information.
Currently, the local locators operate somewhat independently during the exploration phase. Future research will focus on improving real-time communication between these locators and refining end-to-end training techniques to further synchronize local and global feature extraction. Despite these areas for growth, ViMo represents a significant step toward real-time, high-accuracy understanding of long-form video content.
Summary updated