Abstract
Estimating the motion of the camera together with the 3D structure of the scene from a monocular vision system is a complex task that often relies on the so-called scene rigidity assumption. When observing a dynamic environment, this assumption is violated which leads to an ambiguity between the ego-motion of the camera and the motion of the objects. To solve this problem, we present a self-supervised learning framework for 3D object motion field estimation from monocular videos. Our contributions are two-fold. First, we propose a two-stage projection pipeline to explicitly disentangle the camera ego-motion and the object motions with dynamics attention module, called DAM. Specifically, we design an integrated motion model that estimates the motion of the camera and object in the first and second warping stages, respectively, controlled by the attention module through a shared motion encoder. Second, we propose an object motion field estimation through contrastive sample consensus, called CSAC, taking advantage of weak semantic prior (bounding box from an object detector) and geometric constraints (each object respects the rigid body motion model). Experiments on KITTI, Cityscapes, and Waymo Open Dataset demonstrate the relevance of our approach and show that our method outperforms state-of-the-art algorithms for the tasks of self-supervised monocular depth estimation, object motion segmentation, monocular scene flow estimation, and visual odometry.
Introduction
In computer vision, monocular depth estimation—deriving 3D structure from a single 2D video—traditionally relies on the "scene rigidity assumption," which posits that the camera is the only moving element. However, real-world environments are dynamic, containing moving vehicles and pedestrians. This movement creates "motion ambiguity," where the system cannot distinguish between camera ego-motion and object motion, resulting in distorted depth maps and "ghosting" artifacts.
This paper introduces a self-supervised framework that addresses these challenges by simultaneously estimating 3D depth and complex motion. By employing a novel Dynamics Attention Module (DAM) to separate camera movement from object movement and a Contrastive Sample Consensus (CSAC) technique to enforce consistency on moving objects, the authors achieve state-of-the-art results in depth estimation and scene flow without requiring expensive ground-truth labels.
Methodology
The proposed pipeline disentangles conflicting motions through a two-stage process:
Dynamics Attention Module (DAM)
Instead of employing separate, computationally heavy networks for camera and object motion, the model uses a shared encoder. The DAM, a self-attention mechanism, is integrated to allow the model to selectively focus on the background for camera ego-motion or on foreground objects for residual motion. This shared architecture improves efficiency and enables the learning of richer features by modeling the relationship between the two types of motion.
Contrastive Sample Consensus (CSAC)
To refine the often "noisy" or "leaky" motion fields predicted for moving objects, the authors introduce CSAC. This module utilizes weak semantic priors (2D bounding boxes) to group pixels and applies a RANSAC-like iterative sampling method to determine the most representative motion for an object. A contrastive loss is applied to widen the "motion gap" between the object and its background, effectively sharpening object boundaries.
Key Insights
- Shared Motion Encoding: The success of DAM suggests that the features required to understand camera and object movement are deeply related. Using a shared encoder with an attention-based "switch" outperformed dual-encoder setups.
- The Motion Gap: Explicitly penalizing small foreground motions and large background motions via CSAC creates a natural regularization effect. This "gap" helps the model align the motion field with the physical boundaries of objects.
- Rigidity as a Geometric Constraint: By assuming objects like cars are rigid bodies, the model can infer motion for visually featureless areas (e.g., smooth car doors) based on the motion of more textured parts.
Experiments and Results
The framework was evaluated on the KITTI, Cityscapes, and Waymo Open Datasets. Notably, it utilizes "weak semantic priors" (bounding boxes) rather than pixel-perfect segmentation, enhancing its practical applicability.
Key results include:
- Scene Flow: On the KITTI benchmark, the model achieved a 57.8% accuracy gain in estimating the disparity of moving objects (D2-fg) compared to methods lacking semantic priors.
- Depth Estimation: The model recorded the lowest Absolute Relative (AbsRel) error on KITTI (0.111) and Waymo (0.148) among comparable monocular frameworks.
- Visual Odometry: Improved object motion estimation directly translated to more accurate camera path tracking.
Conclusion
While highly effective for rigid objects, the current framework is limited by its rigid body assumption, which can struggle with deformable objects like pedestrians. Future research will focus on extending contrastive learning techniques to handle non-rigid motions, potentially enabling standard cameras to map complex urban environments with high precision.
Summary updated