Multi-motion and Appearance Self-Supervised Moving Object Detection

Fan Yang, Srikrishna Karanam, Meng Zheng, Terrence Chen, Haibin Ling, Ziyan Wu

Abstract

Moving Object Detection (MOD) is a fundamental task in computer vision, traditionally reliant on extensive manual labeling. Self-supervised methods using the Contextual Information Separation (CIS) principle offer a promising alternative but struggle with deformable objects and moving cameras. We present MASNet (Multi-motion and Appearance Self-supervised Network), which addresses these limitations by integrating multi-scale temporal motion cues with a spatial appearance inpainter. Evaluated on the DAVIS dataset, MASNet achieves state-of-the-art performance, significantly outperforming previous self-supervised baselines while maintaining high computational efficiency.

Introduction

Moving Object Detection (MOD) is a cornerstone of computer vision, essential for autonomous driving and surveillance. While supervised methods perform well, they require massive amounts of expensive, hand-labeled data. Recent self-supervised approaches attempt to learn from video data directly, often employing the Contextual Information Separation (CIS) principle. CIS assumes that a moving object’s motion is independent of its background; if a model cannot "guess" the motion inside a masked area based on its surroundings, that area likely contains a moving object.

However, existing CIS-based models face two major limitations. First, they typically rely on single-scale motion (adjacent frames), which fails for deformable objects where different parts move at varying speeds. Second, camera motion introduces background flow that violates the "independent motion" assumption, leading to false detections.

Challenges in self-supervised MOD: multi-scale motion and appearance-based failures.

Methods

To overcome these challenges, we propose MASNet (Multi-motion and Appearance Self-supervised Network). MASNet introduces two novel modules into an adversarial learning pipeline:

  1. Multi-branch Flow Encoding (MFE): Instead of analyzing a single optical flow map, MFE processes multiple maps across different time intervals (e.g., $t \to t+1, t \to t+2, \dots, t \to t+N$). This allows the model to aggregate motion cues from various parts of an object that might appear stationary in one interval but move in another.
  2. Image InPainter (IMIP): This module handles moving cameras by attempting to reconstruct the static appearance of the scene. If the generator suggests a mask in the background, the IMIP can easily reconstruct that area using surrounding textures, signaling that the mask is incorrect. Conversely, if the mask covers a unique moving object, reconstruction fails, confirming a valid detection.

The system utilizes an adversarial training schedule where a Generator creates object masks, while Flow and Image Inpainters attempt to prove those masks wrong by reconstructing the hidden data.

The MASNet architecture, incorporating multi-scale flow and image inpainting.

Results and Discussion

MASNet was evaluated on the DAVIS (Densely Annotated VIdeo Segmentation) dataset using an unsupervised protocol. The model achieved a Jaccard mean score ($J_m$) of 63.2, significantly outperforming the previous CIS baseline of 59.3. Performance gains were particularly notable for deformable objects, with categories like "breakdance" and "cows" seeing improvements of over 10 points.

Beyond accuracy, MASNet is highly efficient. It is approximately 16x faster than previous unsupervised methods like ARP (4.7s vs 74.5s per frame), making it practical for real-world applications.

Conclusion

MASNet demonstrates that motion and appearance are complementary cues that can effectively supervise each other. By looking across multiple temporal scales and using appearance as a regularizer, the model avoids common pitfalls associated with camera noise and complex object deformations. While the current framework relies on post-processing for mask refinement, future work will focus on long-term temporal consistency to maintain object tracking during periods of stasis.

Summary updated

Abstract

In this work, we consider the problem of self-supervised Moving Object Detection (MOD) in video, where no ground truth is involved in both training and inference phases. Recently, an adversarial learning framework is proposed [32] to leverage inherent temporal information for MOD. While showing great promising results, it uses single scale temporal information and may meet problems when dealing with a deformable object under multi-scale motion in different parts. Additional challenges can arise from the moving camera, which results in the failure of the motion independence hypothesis and locally independent background motion. To deal with these problems, we propose a Multimotion and Appearance Self-supervised Network (MASNet) to introduce multi-scale motion information and appearance information of scene for MOD. In particular, a moving object, especially the deformable, usually consists of moving regions at various temporal scales. Introducing multiscale motion can aggregate these regions to form a more complete detection. Appearance information can serve as another cue for MOD when the motion independence is not reliable and for removing false detection in background caused by locally independent background motion. To encode multi-scale motion and appearance, in MASNet we respectively design a multi-branch flow encoding module and an image inpainter module. The proposed modules and MASNet are extensively evaluated on the DAVIS dataset to demonstrate the effectiveness and superiority to state-of-the-art self-supervised methods.
BibTeX
@inproceedings{yang2022multi,
  author = {Fan Yang and Srikrishna Karanam and Meng Zheng and Terrence Chen and Haibin Ling and Ziyan Wu},
  title = {{Multi-motion and Appearance Self-Supervised Moving Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00216},
}
Figure