Long Movie Clip Classification with State-Space Video Models

Md Mohaiminul Islam, Gedas Bertasius

Introduction

Modern video recognition models are predominantly "short-sighted," optimized for processing clips spanning only 5–10 seconds. While this temporal window is sufficient for identifying atomic actions—such as jumping or cutting an onion—it is inadequate for complex, long-form video understanding. Tasks like identifying a director’s cinematic style, discerning evolving character relationships, or predicting movie genres require a model to synthesize information over much longer durations.

The primary technical bottleneck in long-form video processing is the self-attention mechanism inherent in Transformers. While effective at capturing dependencies, its computational cost grows quadratically ($O(L^2)$) relative to the sequence length. As video duration and resolution increase, the number of tokens explodes, leading to prohibitive memory consumption and slow inference. To circumvent this, researchers often rely on "frozen" pre-extracted features, which unfortunately discard the fine-grained spatiotemporal details necessary for deep narrative understanding.

Methodology: The ViS4mer Architecture

To address these limitations, we introduce ViS4mer, a hybrid model designed to process thousands of tokens across extended temporal windows efficiently. ViS4mer replaces the computationally expensive temporal self-attention with Structured State-Space (S4) layers, which operate with linear complexity ($O(L)$). The architecture is divided into two distinct stages:

1. Transformer Encoder (Short-range)

A standard Vision Transformer (ViT) encoder processes individual video frames independently. This stage focuses on extracting high-quality, patch-level spatial features, ensuring that the model maintains the high-resolution precision characteristic of modern Transformers.

2. Multi-scale Temporal S4 Decoder (Long-range)

The core innovation of ViS4mer is its temporal decoder. Instead of using self-attention to link frames over time, the model utilizes S4 layers. To enhance the effectiveness of the S4 decoder, the authors implemented a multi-scale strategy. The decoder consists of multiple blocks that progressively reduce the number of tokens via pooling and decrease channel dimensions. This hierarchical approach allows the model to learn features at varying granularities—from localized movements to long-term narrative arcs—while significantly reducing the total computational load.

Experimental Setup and Datasets

ViS4mer was evaluated on three primary benchmarks designed to test long-range temporal reasoning:

  • Long Video Understanding (LVU): A rigorous benchmark featuring nine tasks, including content understanding (e.g., character relationships), metadata prediction (e.g., movie genre, director), and user engagement (e.g., popularity).
  • Breakfast Dataset: A procedural activity dataset focused on long-range cooking sequences.
  • COIN: A large-scale dataset for comprehensive instructional video analysis, such as DIY projects.

Key Insights

Challenging the Efficiency-Performance Tradeoff

Typically, "efficient" Transformer variants (e.g., Performer or Orthoformer) sacrifice accuracy for speed. ViS4mer breaks this trend, actually outperforming standard Transformers on long-range tasks. Its efficiency allows it to process a higher density of spatiotemporal tokens that would cause standard Transformers to exceed memory limits, leading to better overall representation.

S4 as a Temporal Bridge

The research demonstrates a functional division of labor: self-attention is excellent for spatial feature extraction within a frame, while S4 is superior for "remembering" and reasoning across long temporal sequences. Furthermore, the multi-scale design—pooling tokens as they move deeper into the decoder—is critical for generalization and prevents overfitting on the relatively small datasets common in long-form video research.

Results and Performance

ViS4mer demonstrates significant improvements over existing baselines in both accuracy and efficiency:

  • State-of-the-Art Performance: ViS4mer achieved the best results in 6 out of 9 tasks on the LVU benchmark, outperforming established models like VideoBERT and Object Transformer.
  • Computational Speed: When processing 11,760 spatiotemporal tokens, ViS4mer is 2.63x faster than a comparable Transformer baseline.
  • Memory Efficiency: In high-token scenarios, ViS4mer required only 5.15 GB of GPU memory, compared to 41.38 GB for the standard Transformer—a nearly 8x reduction.
  • Temporal Scaling: Accuracy continued to climb as input video length increased from 1 to 60 seconds, whereas standard models typically plateaued or failed due to data volume.

Limitations and Future Work

While ViS4mer excels at classification and regression, its application to other long-form tasks—such as video summarization, question answering, or video grounding—remains an area for future exploration. Additionally, while 60-second clips represent a significant leap forward, they are still short compared to full-length feature films. Future iterations may involve stacking more hierarchical levels to move toward "Full Movie" understanding and exploring end-to-end pre-training on massive video datasets to further enhance performance.

Summary updated

BibTeX
@inproceedings{islam2022long,
  author = {Md Mohaiminul Islam and Gedas Bertasius},
  title = {{Long Movie Clip Classification with State-Space Video Models}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=1S7vta1WsK},
}
Figure