Abstract
While most modern video understanding models operate on short-range clips, real-world videos are often several minutes long with semantically-consistent segments of variable length. A common approach to process long videos is applying a short-form video model over uniformly sampled clips of fixed temporal length and aggregating the outputs. This approach neglects the underlying nature of long videos since fixed-length clips are often redundant or uninformative. In this paper, we aim to provide a generic and adaptive sampling approach for long-form videos in lieu of the de facto uniform sampling. Viewing videos as semantically-consistent segments, we formulate a task-agnostic, unsupervised, and scalable approach based on Kernel Temporal Segmentation (KTS) for sampling and tokenizing long videos. We evaluate our method on long-form video understanding tasks such as video classification and temporal action localization, showing consistent gains over existing approaches and achieving state-of-the-art performance on long-form video modeling.
Introduction
Modern video AI models excel at analyzing 10-second clips but often struggle with long-form content due to memory and computational constraints. The industry standard for handling long videos is "uniform sampling," which involves dividing a video into fixed-length chunks (e.g., one frame every few seconds). However, this method is fundamentally flawed for three reasons: it creates redundancy in slow scenes, loses critical information in fast-moving scenes, and often cuts through the middle of meaningful events, disrupting the semantic narrative.
While adaptive samplers exist, they are typically "black boxes" trained for specific tasks and lack generalizability. This research proposes using Kernel Temporal Segmentation (KTS) as an unsupervised, adaptive "tokenizer" that breaks videos into semantically consistent segments before sampling, offering a more efficient and task-agnostic alternative.
The KTS Adaptive Tokenizer
The core innovation lies in treating a video as a sequence of semantically consistent segments rather than arbitrary frames. By repurposing the KTS algorithm as a modern tokenizer, the authors create a workflow that respects the natural boundaries of actions within a video.
As shown in the pipeline above, the process involves four key steps:
- Feature Extraction: A lightweight pre-trained model (e.g., ResNet) extracts visual features from each frame.
- Change Point Detection: KTS computes a similarity matrix to identify "change points" where visual content shifts significantly.
- Adaptive Sampling: The video is split into $m$ segments of varying lengths based on these change points, and $k$ frames are uniformly sampled from within each segment.
- Downstream Processing: These representative frames are fed into high-performance models for tasks like classification or localization.
Experimental Results
The researchers evaluated the KTS approach against three major benchmarks: the Breakfast Dataset (cooking activities), LVU (Long-form Video Understanding from movies), and ActivityNet (daily activities).
Performance on LVU Benchmark
On the LVU benchmark, which includes diverse tasks such as predicting movie genres or directors, the KTS-based approach consistently outperformed uniform sampling. The gains were particularly notable in "scene prediction," where accuracy increased by 12.79%.
Efficiency and Localization
- Breakfast Dataset: The model achieved a new state-of-the-art (SOTA) accuracy of 89.86% while using 16x fewer frames than the baseline. This demonstrates that the quality of frame selection is often more critical than the quantity.
- Temporal Action Localization (TAL): On ActivityNet, KTS proved superior for finding action boundaries. Because KTS naturally respects visual shifts, it avoids the "blurring" of action starts and ends common in uniform sampling, leading to higher precision even when the data budget is tight.
Conclusion
By framing video segmentation as an "adaptive tokenization" problem, this work demonstrates that unsupervised change-point detection can significantly improve long-form video understanding. KTS acts as a plug-and-play component that enhances efficiency and accuracy across multiple tasks without requiring task-specific training for the sampler itself.
Future work may involve developing a learnable version of KTS that can be trained end-to-end, potentially allowing the model to decide the optimal number of segments based on the complexity of the video content.
Summary updated