SIGMA: Sinkhorn-Guided Masked Video Modeling

Mohammadreza Salehi, Michael Dorkenwald, Fida Mohammad Thoker, Efstratios Gavves, Cees G. M. Snoek, Yuki M. Asano

Abstract

Masked Video Modeling (MVM) has emerged as a powerful paradigm for self-supervised representation learning in computer vision. However, traditional MVM approaches often rely on pixel-level reconstruction, which emphasizes low-level visual details over high-level semantic understanding. This paper presents SIGMA (Sinkhorn-Guided Masked Video Modeling), a framework designed to shift the learning objective from raw pixels to a semantic feature space. By leveraging the Sinkhorn-Knopp algorithm to solve an optimal transport problem, SIGMA clusters video features into meaningful prototypes, preventing representation collapse and forcing the model to capture robust spatial and temporal concepts. SIGMA achieves state-of-the-art performance across ten benchmarks, demonstrating superior generalization and semantic depth.

Introduction

In the field of computer vision, the "fill-in-the-blanks" approach of Masked Video Modeling (MVM) has become a standard for training autonomous systems. By hiding portions of a video and requiring a model to predict the missing content, researchers can train models without manual labels. Despite its success, current MVM implementations, such as VideoMAE, primarily focus on reconstructing raw pixels. This task forces models to allocate significant computational capacity to low-level signals like texture and brightness, which may not be essential for understanding complex actions or object dynamics.

Transitioning from pixel prediction to feature prediction presents a significant challenge: representation collapse. Without a strong regularizer, a model and its target network can reach a trivial solution where they output constant values, satisfying the loss function without learning useful features. SIGMA addresses this by introducing a mathematical bottleneck grounded in optimal transport theory.

Methodology

Sinkhorn-Guided Clustering

The core innovation of SIGMA is the replacement of pixel targets with cluster assignments. The framework utilizes a projection network (a Multi-Layer Perceptron) to embed video patches into a latent feature space. To ensure these features are meaningful and to prevent collapse, SIGMA treats the assignment of features to "prototypes" (learnable centroids) as an optimal transport problem.

Using the Sinkhorn-Knopp algorithm, SIGMA enforces an equipartition constraint. This constraint requires the model to distribute video features evenly across all available prototypes, preventing the model from "cheating" by collapsing all inputs into a single category. This process effectively forces the model to categorize the visual world into a diverse set of semantic concepts.

Symmetric Prediction and Knowledge Distillation

SIGMA employs a symmetric prediction strategy where the main video model predicts the cluster assignments generated by the projection network, and vice versa. This cross-prediction mechanism ensures that both components converge on a high-entropy, informative feature space. Furthermore, the authors demonstrate that SIGMA can incorporate frozen, pre-trained image models (such as DINO) as the projection network. This allows the video model to inherit rich spatial knowledge from image-based datasets while focusing its learning capacity on temporal dynamics.

Experimental Results

Action Recognition and Semantic Depth

SIGMA was evaluated on major benchmarks including Kinetics-400 (K400) and Something-Something V2 (SSv2). A key highlight of the results is the frozen evaluation performance. When the backbone is frozen and only a linear classifier is trained, SIGMA with DINO features achieved 47.5% accuracy on K400, more than doubling the 20.7% achieved by the pixel-based VideoMAE. This suggests that SIGMA learns representations that are inherently more semantic and useful for downstream tasks without further fine-tuning.

Generalization and Segmentation

On the SEVERE benchmark, which evaluates model performance under domain shifts and limited data, SIGMA outperformed existing methods, indicating high robustness. In unsupervised video object segmentation (DAVIS and YTVOS), SIGMA produced cleaner and more temporally consistent object masks than its predecessors, confirming that the model develops a sophisticated understanding of object boundaries and motion.

Conclusion and Future Work

SIGMA represents a significant advancement in self-supervised video learning by moving beyond the limitations of pixel-level reconstruction. By utilizing optimal transport to guide semantic clustering, the framework produces representations that are both robust and conceptually rich. While current research was limited to ViT-Base architectures due to computational constraints, future work may explore scaling SIGMA to larger models and extending the Sinkhorn-guided approach to multi-modal settings, such as integrating audio or text to further refine the learned prototypes.

Summary updated

BibTeX
@inproceedings{salehi2024sigma,
  author = {Mohammadreza Salehi and Michael Dorkenwald and Fida Mohammad Thoker and Efstratios Gavves and Cees G. M. Snoek and Yuki M. Asano},
  title = {{SIGMA: Sinkhorn-Guided Masked Video Modeling}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=fUPuoNfb9P},
}
Figure