Abstract
Salient motion segmentation—identifying independently moving objects in video—is often confounded by camera motion, which induces 2D movement in static background elements. This paper presents EpO-Net, a deep learning architecture that explicitly incorporates 3D geometric constraints to improve motion saliency. By leveraging dense trajectories and epipolar distances calculated via trifocal tensors, the model distinguishes between camera-induced motion and true independent motion. A novel input-dropout training strategy ensures robust feature fusion. EpO-Net achieves state-of-the-art performance on the DAVIS-2016, DAVIS-2017, and Segtrackv2 benchmarks, significantly reducing false detections on static objects.
Introduction
In computer vision, distinguishing independently moving objects from a moving background is a fundamental challenge. Standard deep learning models often rely on 2D patterns in optical flow, which can lead to "false positives" where prominent static objects (like buildings or trees) are mistakenly labeled as moving due to camera ego-motion. The core issue is that deep networks do not inherently learn the 3D geometric constraints of the physical world from raw pixels alone. This work addresses this gap by explicitly providing geometric cues to the network.
Methodology
The proposed EpO-Net pipeline integrates traditional multiview geometry with modern deep learning through several key components:
- Dense Trajectories and Epipolar Constraints: The system tracks pixels across multiple frames to form trajectories. Using the trifocal tensor, it calculates epipolar distances, which measure how much a pixel's path deviates from the expected path of a static background point. High deviation serves as a strong indicator of independent motion.
- Architecture: The network uses an encoder-decoder structure (similar to U-Net) that takes both raw optical flow and calculated epipolar maps as inputs. This allows the model to combine local motion patterns with global geometric constraints.
- Input-Dropout Training: To prevent the model from over-relying on potentially noisy geometric maps, an "input-dropout" strategy is employed during training. By randomly zeroing out input channels, the network is forced to learn robust, complementary features from both flow and geometry.
- EpO-Net+: The full model fuses motion results with appearance features from a pre-trained DeepLab model and utilizes a ConvGRU to maintain temporal consistency across video frames.
Experimental Results
The model was evaluated on several key benchmarks, including DAVIS-2016, DAVIS-2017, and Segtrackv2. To facilitate training, the authors introduced the RBSF (Real Background, Synthetic Foreground) dataset, which combines real-world background footage with synthetic foreground objects.
Performance Comparison
On the DAVIS-2016 benchmark, the motion-only version of EpO-Net outperformed the previous state-of-the-art (Mp-Net) by 5.2% in mean Intersection over Union (IoU). The full EpO-Net+ model achieved a mean IoU of 80.6%, surpassing other prominent methods such as LVO and AGS.
Key Insights
- Geometric Efficiency: Explicitly calculating geometric constraints is more data-efficient than learning them; EpO-Net achieved superior results with 20k training frames compared to 45k for competitors.
- Robust Fusion: The input-dropout strategy ensures that the model remains effective even when one data source (e.g., geometry on non-rigid backgrounds) is unreliable.
Conclusion
EpO-Net demonstrates that integrating 3D geometric principles into deep learning architectures significantly improves salient motion segmentation. While the current model assumes a rigid world for its geometric constraints, future work could explore non-rigid motion models to handle complex backgrounds like flowing water or foliage. The success of the input-dropout strategy suggests a promising path for multi-modal sensor fusion in computer vision tasks.
Summary updated