Abstract
Moving Object Detection (MOD) is a fundamental task in computer vision, traditionally reliant on extensive manual labeling. Self-supervised methods using the Contextual Information Separation (CIS) principle offer a promising alternative but struggle with deformable objects and moving cameras. We present MASNet (Multi-motion and Appearance Self-supervised Network), which addresses these limitations by integrating multi-scale temporal motion cues with a spatial appearance inpainter. Evaluated on the DAVIS dataset, MASNet achieves state-of-the-art performance, significantly outperforming previous self-supervised baselines while maintaining high computational efficiency.
Introduction
Moving Object Detection (MOD) is a cornerstone of computer vision, essential for autonomous driving and surveillance. While supervised methods perform well, they require massive amounts of expensive, hand-labeled data. Recent self-supervised approaches attempt to learn from video data directly, often employing the Contextual Information Separation (CIS) principle. CIS assumes that a moving object’s motion is independent of its background; if a model cannot "guess" the motion inside a masked area based on its surroundings, that area likely contains a moving object.
However, existing CIS-based models face two major limitations. First, they typically rely on single-scale motion (adjacent frames), which fails for deformable objects where different parts move at varying speeds. Second, camera motion introduces background flow that violates the "independent motion" assumption, leading to false detections.
Methods
To overcome these challenges, we propose MASNet (Multi-motion and Appearance Self-supervised Network). MASNet introduces two novel modules into an adversarial learning pipeline:
- Multi-branch Flow Encoding (MFE): Instead of analyzing a single optical flow map, MFE processes multiple maps across different time intervals (e.g., $t \to t+1, t \to t+2, \dots, t \to t+N$). This allows the model to aggregate motion cues from various parts of an object that might appear stationary in one interval but move in another.
- Image InPainter (IMIP): This module handles moving cameras by attempting to reconstruct the static appearance of the scene. If the generator suggests a mask in the background, the IMIP can easily reconstruct that area using surrounding textures, signaling that the mask is incorrect. Conversely, if the mask covers a unique moving object, reconstruction fails, confirming a valid detection.
The system utilizes an adversarial training schedule where a Generator creates object masks, while Flow and Image Inpainters attempt to prove those masks wrong by reconstructing the hidden data.
Results and Discussion
MASNet was evaluated on the DAVIS (Densely Annotated VIdeo Segmentation) dataset using an unsupervised protocol. The model achieved a Jaccard mean score ($J_m$) of 63.2, significantly outperforming the previous CIS baseline of 59.3. Performance gains were particularly notable for deformable objects, with categories like "breakdance" and "cows" seeing improvements of over 10 points.
Beyond accuracy, MASNet is highly efficient. It is approximately 16x faster than previous unsupervised methods like ARP (4.7s vs 74.5s per frame), making it practical for real-world applications.
Conclusion
MASNet demonstrates that motion and appearance are complementary cues that can effectively supervise each other. By looking across multiple temporal scales and using appearance as a regularizer, the model avoids common pitfalls associated with camera noise and complex object deformations. While the current framework relies on post-processing for mask refinement, future work will focus on long-term temporal consistency to maintain object tracking during periods of stasis.
Summary updated