EpO-Net: Exploiting Geometric Constraints on Dense Trajectories for Motion Saliency

Muhammad Faisal, Ijaz Akhter, Mohsen Ali, Richard I. Hartley

Abstract

Salient motion segmentation aims to isolate independently moving objects in video sequences, a task complicated by dynamic camera motion. While deep learning models have shown promise, they often fail to internalize the geometric principles of the physical world, leading to false positives on static but salient background elements. We present EpO-Net, a framework that explicitly incorporates multiview geometric constraints—specifically epipolar distances derived from dense trajectories—into a deep learning architecture. By fusing these geometric priors with optical flow and appearance features, EpO-Net achieves state-of-the-art performance on benchmarks like DAVIS and Segtrackv2, demonstrating superior robustness and data efficiency.

Introduction

The primary objective of salient motion segmentation is to identify objects moving independently of the background. In scenarios with a moving camera, this task becomes significantly more complex as every pixel exhibits a non-zero motion vector. Traditional deep learning approaches typically attempt to learn the distinction between foreground and background motion directly from raw optical flow or appearance data. However, these networks often struggle to "discover" underlying geometric laws autonomously. Consequently, they frequently produce false positives, misidentifying prominent static objects (e.g., parked cars or buildings) as moving due to their visual saliency or relative motion caused by camera perspective.

Methodology

The proposed EpO-Net architecture utilizes a two-stage approach to integrate geometric reasoning into the segmentation process.

Geometric Feature Extraction

The process begins by converting optical flow into dense trajectories, tracking pixels across multiple frames. Using trifocal tensors calculated via RANSAC and a six-point algorithm, the system computes "epipolar distances" for these trajectories. In a rigid scene, the movement of a point across frames is constrained by epipolar geometry. Significant deviations from these constraints (high epipolar distance) indicate that a point is likely moving independently of the background.

EpO-Net Architecture

The core network is a U-Net-style encoder-decoder that processes a three-channel "motion image." This input consists of two channels for optical flow (X-Y displacement) and one channel dedicated to the calculated epipolar distance. To ensure the model does not over-rely on any single, potentially noisy modality, the authors implement "input-dropout," where channels are randomly zeroed out during training. This forces the network to learn robust, complementary features.

EpO-Net+ (Fusion)

To address non-rigid backgrounds (such as moving water) where geometric constraints may falter, the motion-saliency map is fused with appearance features from a pre-trained DeepLab model. This fusion is managed by a bidirectional Convolutional Gated Recurrent Unit (ConvGRU), allowing the model to leverage both long-term geometric consistency and short-term appearance cues.

Experiments and Results

The model was evaluated on the DAVIS-2016, DAVIS-2017, and Segtrack-v2 datasets. To provide a robust training ground for geometric constraints, the authors also introduced the RBSF (Real Background and Synthetic Foreground) dataset, which combines real-world background videos with synthetic foreground objects.

Key Performance Metrics

On the DAVIS-2016 benchmark, the motion-only version of EpO-Net outperformed the previous state-of-the-art motion-only network, Mp-Net, by 5.2% in mean IoU (0.752 vs 0.700). The full fusion model, EpO-Net+, achieved a mean IoU of 80.6%, surpassing contemporary methods such as LVO and AGS.

Qualitative comparison of EpO-Net against state-of-the-art methods.

Qualitative Analysis

Qualitative results highlight the strength of the geometric prior. As shown in the comparison above, EpO-Net successfully ignores large, high-contrast static objects—such as the background car—that typically trigger false detections in other frameworks. This demonstrates that providing epipolar distance as an explicit input acts as a powerful regularizer, helping the network distinguish true independent motion from camera-induced parallax.

Discussion and Conclusion

EpO-Net demonstrates that while deep learning is a powerful tool, explicitly providing physical priors like epipolar geometry can significantly enhance performance and data efficiency. By using a strong geometric prior, the model required only 20,000 frames for training, compared to the 45,000 frames used by previous leading models.

Despite its success, the model faces limitations in environments with highly non-rigid backgrounds (e.g., heavy rain) and remains dependent on the quality of the underlying optical flow. Future research may focus on integrating geometric constraint estimation directly into an end-to-end learning process to improve speed and robustness.

Summary updated

Abstract

The existing approaches for salient motion segmentation are unable to explicitly learn geometric cues and often give false detections on prominent static objects. We exploit multiview geometric constraints to avoid such shortcomings. To handle the nonrigid background like a sea, we also propose a robust fusion mechanism between motion and appearance-based features. We find dense trajectories, covering every pixel in the video, and propose trajectory-based epipolar distances to distinguish between background and foreground regions. Trajectory epipolar distances are dataindependent and can be readily computed given a few features' correspondences between the images. We show that by combining epipolar distances with optical flow, a powerful motion network can be learned. Enabling the network to leverage both of these features, we propose a simple mechanism, we call input-dropout. Comparing the motion-only networks, we outperform the previous state of the art on DAVIS-2016 dataset by 5.2% in the mean IoU score. By robustly fusing our motion network with an appearance network using the input-dropout mechanism, we also outperform the previous methods on DAVIS-2016, 2017 and Segtrackv2 dataset.
BibTeX
@inproceedings{faisal2020epo,
  author = {Muhammad Faisal and Ijaz Akhter and Mohsen Ali and Richard I. Hartley},
  title = {{EpO-Net: Exploiting Geometric Constraints on Dense Trajectories for Motion Saliency}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093589},
}
Figure