Supervoxel Attention Graphs for Long-Range Video Modeling

Yang Wang, Gedas Bertasius, Tae-Hyun Oh, Abhinav Gupta, Minh Hoai, Lorenzo Torresani

Introduction

Video understanding remains one of the most computationally demanding tasks in computer vision. While 3D Convolutional Neural Networks (3D CNNs) excel at processing short video clips, they are fundamentally limited when tasked with long-range temporal modeling. The high dimensionality of video data—millions of pixels across space and time—creates a memory bottleneck that restricts the temporal receptive field of standard architectures.

Existing attempts to bridge this gap, such as Non-Local Networks, attempt to model global dependencies by computing pixel-to-pixel relationships. However, these methods are computationally prohibitive for videos exceeding a few seconds. Conversely, Recurrent Neural Networks (RNNs) often over-compress frames into single vectors, sacrificing the fine-grained spatial structure necessary for complex action recognition. The Supervoxel Attention Graph (SVAG) framework addresses these challenges by representing video as a sparse, semantic graph, enabling efficient long-range modeling without losing spatial detail.

Methodology: Semantic Supervoxel Representation

The core innovation of SVAG is the transition from dense pixel grids to sparse "supervoxels." Instead of treating every pixel independently, the model groups pixels into spatiotemporal "tubes" that track objects or parts across frames.

Comparison between standard grid-based partitioning (a) and semantic supervoxel generation (b). Supervoxels track objects through time, providing a more meaningful representation for graph nodes.

As shown in the figure above, standard grid partitioning (a) is agnostic to the scene's content. In contrast, the SVAG pipeline uses features from a semantic segmentation network (e.g., UPerNet) alongside color and spatial coordinates to generate object-aware supervoxels (b). This ensures that the resulting graph nodes correspond to meaningful entities in the video.

The SVAG Architecture

Once supervoxels are generated, they are treated as nodes in a graph. Features are pooled for each node from a 3D CNN backbone (such as ResNet-101 or CSN). The network then processes this graph through two parallel branches:

  1. Relational Branch: This branch utilizes Graph Convolutional Networks (GCNs) to facilitate message passing between nodes. It employs Self-Attention Graph Pooling (SAGPool) to hierarchically identify and focus on the most salient spatiotemporal regions while discarding less relevant nodes.
  2. Local Branch: This branch processes each node independently, ensuring that specific local motion and appearance cues are preserved throughout the network.

A critical design choice is the Source-Preserve mechanism. Rather than completely deleting nodes discarded during pooling, the model retains them as context for message passing. This prevents the premature loss of background information that might be vital for understanding the overall action context.

Results and Key Insights

The SVAG model was evaluated on the Charades and Something-Something-v1 datasets, both of which require understanding complex, long-form human activities.

Spontaneous Attention and Sparsity

One of the most striking findings is that the model requires very little data to represent a complex scene. The authors demonstrate that just 16 supervoxels per second are sufficient for high-accuracy action recognition. Furthermore, the model's attention mechanism naturally learns to prioritize relevant regions.

Visualization of spontaneous attention in SVAG. The model naturally learns to focus on salient regions like humans and manipulated objects (highlighted in red) without explicit supervision.

As illustrated in the attention maps above, the model focuses its computational resources on the person and the objects they interact with. This "spontaneous attention" emerges without any direct supervision or hand-crafted rules, proving the effectiveness of the graph-based relational reasoning.

Performance and Efficiency

SVAG achieves state-of-the-art results on the Charades dataset with a 44.1 mAP, significantly outperforming Non-Local Networks (37.5) and Long-Term Feature Banks (42.5). Beyond accuracy, the efficiency gains are substantial: the SVAG classification head is approximately 90x faster and uses 20x less memory during training compared to Non-Local alternatives.

Discussion and Limitations

While SVAG offers a powerful alternative for long-range modeling, some challenges remain. The model is sensitive to temporal noise; connecting nodes across excessively large time gaps can sometimes degrade performance. To mitigate this, the authors implemented temporal constraints to focus connections within a 10-30 second window. Additionally, the current pipeline relies on a pre-trained segmentation network for supervoxel generation. Future iterations could explore end-to-end learning of these supervoxels to further streamline the architecture.

In conclusion, Supervoxel Attention Graphs provide a scalable, memory-efficient, and semantically rich framework for video understanding, paving the way for more sophisticated modeling of long-form visual content.

Summary updated

Abstract

A significant challenge in video understanding is posed by the high dimensionality of the input, which induces large computational cost and high memory footprints. Deep convolutional models operating on video apply pooling and striding to reduce feature dimensionality and to increase the receptive field. However, despite these strategies, modern approaches cannot effectively leverage spatiotemporal structure over long temporal extents. In this paper we introduce an approach that reduces a video of 10 seconds to a sparse graph of only 160 feature nodes such that efficient inference in this graph produces state-of-the-art accuracy on challenging action recognition datasets. The nodes of our graph are semantic supervoxels that capture the spatiotemporal structure of objects and motion cues in the video, while edges between nodes encode spatiotemporal relations and feature similarity. We demonstrate that a shallow network that interleaves graph convolution and graph pooling on this compact representation implements an effective mechanism of relational reasoning yielding strong recognition results on both Charades and Something-Something.
BibTeX
@inproceedings{wang2021supervoxel,
  author = {Yang Wang and Gedas Bertasius and Tae-Hyun Oh and Abhinav Gupta and Minh Hoai and Lorenzo Torresani},
  title = {{Supervoxel Attention Graphs for Long-Range Video Modeling}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00020},
}
Figure