Modeling Video as Stochastic Processes for Fine-Grained Video Representation Learning

Heng Zhang, Daqing Liu, Qi Zheng, Bing Su

Introduction

Fine-grained video representation learning is essential for understanding complex, subtle human actions. Traditional methods often rely on discrete frame-level comparisons or labor-intensive alignment between different video sequences. This paper introduces Video as Stochastic Processes (VSP), a novel framework that treats video sequences as continuous evolutions within a latent embedding space. By modeling these sequences as "Brownian bridges," the authors enable machines to capture the inherent temporal dynamics and "intent" of an action with minimal supervision.

The Problem: Beyond Discrete Alignment

Existing approaches to video understanding generally fall into two categories: Video Alignment and View Relevance. Video Alignment requires explicit labels to synchronize two videos of the same action, which is difficult to scale. View Relevance uses self-supervised pretext tasks, such as predicting temporal order, but often fails to capture the underlying physical logic of the motion.

Comparison of Video Alignment, View Relevance, and the proposed Process Agreement approach.

The fundamental limitation is that these models treat frames as isolated points rather than parts of a coherent, goal-oriented process. VSP addresses this by focusing on "Process Agreement," where the model learns the internal consistency of a single video's evolution.

Methodology: The Brownian Bridge Framework

The core innovation of VSP is modeling video phases as Brownian bridges. In mathematics, a Brownian bridge is a stochastic process where the start and end points are fixed. The uncertainty of the particle's position is zero at the endpoints and reaches its maximum at the midpoint.

The VSP Pipeline

The framework consists of three main stages:

  1. Encoding: Video frames are processed through a spatio-temporal encoder (typically a ResNet-50 combined with a 3-layer Transformer) to generate latent embeddings.
  2. Brownian Bridge Construction: The model identifies triplets of frames—a start ($z_A$), an internal point ($z_t$), and an end ($z_T$). It assumes that the transition density between these points follows a time-variant Gaussian distribution.
  3. Process-based Contrastive Loss (PCL): This loss function "pulls" intermediate frame embeddings toward the predicted path of a Brownian bridge while "pushing" away irrelevant or out-of-process frames.
The VSP framework: (a) modeling video as a stochastic process, (b) constructing the Brownian bridge, and (c) the process contrastive training objective.

Supervision Levels

VSP is versatile and can be adapted based on available data:

  • Self-supervised (VSP): Samples triplets randomly from raw video.
  • Weakly-supervised (VSP-P): Uses known phase boundaries as bridge endpoints.
  • Fully-supervised (VSP-F): Incorporates frame-level labels to further cluster similar sub-actions.

Experimental Results

The model was rigorously evaluated on datasets requiring high temporal precision, including PennAction, Pouring, IKEA ASM, and FineGym.

State-of-the-Art Performance

VSP consistently outperformed previous state-of-the-art methods like TCC, LAV, and CARL across all benchmarks. Notably, on the PennAction dataset, VSP achieved a "Progress" score of 0.923, a significant jump from the previous best of 0.789. This metric demonstrates the model's superior ability to track the linear progression of an action over time.

Efficiency and Generalization

One of the most compelling results is VSP's data efficiency. The model achieved performance comparable to fully supervised benchmarks using only 10% of the labeled data. Furthermore, when pre-trained on the large-scale Kinetics-400 dataset without labels, the learned representations generalized effectively to various downstream tasks, proving that the stochastic process model captures universal features of motion.

Discussion and Future Work

While VSP represents a significant step forward, it relies on the assumption that video transitions are relatively smooth. Highly erratic motions or videos with frequent cuts may challenge the Brownian bridge model. Future research could explore more complex stochastic processes, such as Ornstein-Uhlenbeck processes, to model non-linear or multi-modal temporal dynamics.

In conclusion, by shifting the focus from discrete frames to continuous processes, VSP provides a robust and efficient framework for fine-grained video understanding, paving the way for more sophisticated temporal modeling in computer vision.

Summary updated

Abstract

A meaningful video is semantically coherent and changes smoothly. However, most existing fine-grained video representation learning methods learn frame-wise features by aligning frames across videos or exploring relevance between multiple views, neglecting the inherent dynamic process of each video. In this paper, we propose to learn video representations by modeling Video as Stochastic Processes (VSP) via a novel process-based contrastive learning framework, which aims to discriminate between video processes and simultaneously capture the temporal dynamics in the processes. Specifically, we enforce the embeddings of the frame sequence of interest to approximate a goal-oriented stochastic process, i.e., Brownian bridge, in the latent space via a process-based contrastive loss. To construct the Brownian bridge, we adapt specialized sampling strategies under different annotations for both self-supervised and weakly-supervised learning. Experimental results on four datasets show that VSP stands as a state-of-the-art method for various video understanding tasks, including phase progression, phase classification, and frame retrieval. Code is available at https://github.com/hengRUC/VSP.
BibTeX
@inproceedings{zhang2023modeling,
  author = {Heng Zhang and Daqing Liu and Qi Zheng and Bing Su},
  title = {{Modeling Video as Stochastic Processes for Fine-Grained Video Representation Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00221},
}
Figure