Abstract
Salient motion segmentation aims to isolate independently moving objects in video sequences, a task complicated by dynamic camera motion. While deep learning models have shown promise, they often fail to internalize the geometric principles of the physical world, leading to false positives on static but salient background elements. We present EpO-Net, a framework that explicitly incorporates multiview geometric constraints—specifically epipolar distances derived from dense trajectories—into a deep learning architecture. By fusing these geometric priors with optical flow and appearance features, EpO-Net achieves state-of-the-art performance on benchmarks like DAVIS and Segtrackv2, demonstrating superior robustness and data efficiency.
Introduction
The primary objective of salient motion segmentation is to identify objects moving independently of the background. In scenarios with a moving camera, this task becomes significantly more complex as every pixel exhibits a non-zero motion vector. Traditional deep learning approaches typically attempt to learn the distinction between foreground and background motion directly from raw optical flow or appearance data. However, these networks often struggle to "discover" underlying geometric laws autonomously. Consequently, they frequently produce false positives, misidentifying prominent static objects (e.g., parked cars or buildings) as moving due to their visual saliency or relative motion caused by camera perspective.
Methodology
The proposed EpO-Net architecture utilizes a two-stage approach to integrate geometric reasoning into the segmentation process.
Geometric Feature Extraction
The process begins by converting optical flow into dense trajectories, tracking pixels across multiple frames. Using trifocal tensors calculated via RANSAC and a six-point algorithm, the system computes "epipolar distances" for these trajectories. In a rigid scene, the movement of a point across frames is constrained by epipolar geometry. Significant deviations from these constraints (high epipolar distance) indicate that a point is likely moving independently of the background.
EpO-Net Architecture
The core network is a U-Net-style encoder-decoder that processes a three-channel "motion image." This input consists of two channels for optical flow (X-Y displacement) and one channel dedicated to the calculated epipolar distance. To ensure the model does not over-rely on any single, potentially noisy modality, the authors implement "input-dropout," where channels are randomly zeroed out during training. This forces the network to learn robust, complementary features.
EpO-Net+ (Fusion)
To address non-rigid backgrounds (such as moving water) where geometric constraints may falter, the motion-saliency map is fused with appearance features from a pre-trained DeepLab model. This fusion is managed by a bidirectional Convolutional Gated Recurrent Unit (ConvGRU), allowing the model to leverage both long-term geometric consistency and short-term appearance cues.
Experiments and Results
The model was evaluated on the DAVIS-2016, DAVIS-2017, and Segtrack-v2 datasets. To provide a robust training ground for geometric constraints, the authors also introduced the RBSF (Real Background and Synthetic Foreground) dataset, which combines real-world background videos with synthetic foreground objects.
Key Performance Metrics
On the DAVIS-2016 benchmark, the motion-only version of EpO-Net outperformed the previous state-of-the-art motion-only network, Mp-Net, by 5.2% in mean IoU (0.752 vs 0.700). The full fusion model, EpO-Net+, achieved a mean IoU of 80.6%, surpassing contemporary methods such as LVO and AGS.
Qualitative Analysis
Qualitative results highlight the strength of the geometric prior. As shown in the comparison above, EpO-Net successfully ignores large, high-contrast static objects—such as the background car—that typically trigger false detections in other frameworks. This demonstrates that providing epipolar distance as an explicit input acts as a powerful regularizer, helping the network distinguish true independent motion from camera-induced parallax.
Discussion and Conclusion
EpO-Net demonstrates that while deep learning is a powerful tool, explicitly providing physical priors like epipolar geometry can significantly enhance performance and data efficiency. By using a strong geometric prior, the model required only 20,000 frames for training, compared to the 45,000 frames used by previous leading models.
Despite its success, the model faces limitations in environments with highly non-rigid backgrounds (e.g., heavy rain) and remains dependent on the quality of the underlying optical flow. Future research may focus on integrating geometric constraint estimation directly into an end-to-end learning process to improve speed and robustness.
Summary updated