Unsupervised Space-Time Network for Temporally-Consistent Segmentation of Multiple Motions

Etienne Meunier, Patrick BouthemyCVPR 2023 · DOI

Introduction

Motion segmentation is a fundamental computer vision task, crucial for applications ranging from autonomous navigation to action recognition. Despite the success of deep learning, existing methods often struggle with two primary challenges: the high cost of manual video annotation and the prevalence of temporal flickering in segmentations produced by frame-independent processing. This paper presents a fully unsupervised 3D convolutional network designed to discover and segment multiple moving objects by leveraging the joint spatio-temporal evolution of motion.

Methodology: Spatio-Temporal Learning

The proposed framework utilizes a 3D U-Net architecture that processes sub-volumes of optical flow. Rather than analyzing single frames or pairs, the network takes a triplet of successive optical flow fields ($f_{t-\tau}, f_t, f_{t+\tau}$) as input and predicts a corresponding triplet of segmentation masks ($m_{t-\tau}, m_t, m_{t+\tau}$).

Overview of the proposed 3D architecture and temporal consistency loss.

The training is entirely unsupervised, guided by a dual-component loss function:

  1. Spatio-Temporal Parametric Motion Models: For each predicted segment, the model estimates parameters (affine or quadratic) that describe the motion across the three-frame window. A reconstruction loss ($L_r$) ensures these models accurately fit the observed optical flow.
  2. Temporal Consistency Loss ($L_c$): This term enforces stability by penalizing abrupt changes in segmentation masks between frames. To handle occlusions, a masking heuristic ignores regions with high temporal flow variance.

Experimental Evaluation

The model was trained exclusively on the DAVIS2016 training set without ground-truth labels. Its generalization capability was evaluated across three benchmarks: DAVIS2016 (primary object segmentation), SegTrackV2 (multiple objects, low resolution), and FBMS59 (long sequences).

Key Results and Insights

The framework achieves state-of-the-art unsupervised performance, reaching a Jaccard score ($J$) of 73.2 on DAVIS2016. This outperforms other motion-only unsupervised methods and rivals complex models trained on synthetic data.

Qualitative results across various datasets demonstrating the segmentation of multiple moving objects.

Key findings include: * Temporal Stability: Ablation studies highlight the criticality of the temporal consistency loss; removing $L_c$ causes the $J$ score to drop from 73.2 to 33.1. * Efficiency: The lightweight architecture enables real-time inference at 114 fps on a P100 GPU. * Motion as a Sufficient Signal: By relying solely on optical flow, the method demonstrates that temporal motion evolution is a rich enough signal for object discovery without needing RGB appearance features. * Amodal Segmentation: The model can maintain segmentations for objects that momentarily stop moving, thanks to the temporal linkage constraints.

Conclusion and Future Work

While the model excels at segmenting all moving entities, it sometimes faces a "philosophical" mismatch with VOS benchmarks that prioritize only the "primary" object. Future work may involve more sophisticated occlusion reasoning and the use of latent representations to automatically merge or split segments based on complex behavioral patterns.

Summary updated

BibTeX
@inproceedings{meunier2023unsupervised,
  author = {Etienne Meunier and Patrick Bouthemy},
  title = {{Unsupervised Space-Time Network for Temporally-Consistent Segmentation of Multiple Motions}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02120},
}
Figure