What to Select: Pursuing Consistent Motion Segmentation from Multiple Geometric Models

Yangbangyan Jiang, Qianqian Xu, Ke Ma, Zhiyong Yang, Xiaochun Cao, Qingming Huang

Abstract

Motion segmentation is a fundamental task in computer vision, yet real-world complexities like perspective effects and occlusions often render single geometric models insufficient. This paper introduces a novel geometric-model-fusion framework that moves beyond simple consensus-seeking. By employing a multiplicative decomposition strategy and enforcing $k$-block diagonal structural consistency across multiple geometric models (affine, homography, and fundamental), the proposed method successfully filters out unreliable affinities to produce highly accurate, unified segmentations of moving objects in complex video scenes.

The Challenge of Multi-Model Fusion

Motion segmentation involves grouping feature point trajectories in a video into distinct moving objects. However, the diversity of real-world scenes—ranging from tiny objects to significant perspective shifts—makes it nearly impossible for a single geometric model, such as the standard affine model, to capture all motions accurately.

While recent research has shifted toward "model fusion" to combine the strengths of multiple models, most existing methods only seek consensus late in the pipeline, typically within the spectral embedding space. This delay often fails to filter out "harmful" information, such as false-positive connections between different objects present in raw affinity matrices. The core challenge lies in automatically selecting valuable patterns across multiple models while discarding noise.

Methodology: Structural Pursuit via Multiplicative Decomposition

The proposed framework pursues Consistent Motion Segmentation with Structural Pursuit through a multi-step process:

  1. Multi-Model Affinity Construction: Multiple affinity matrices ($A^{(v)}$) are generated using three classic geometric models: Affine, Homography, and Fundamental. These are constructed using Sampson distance and the Ordered Residual Kernel (ORK).
  2. Multiplicative Decomposition: The central innovation is the decomposition of each model's affinity matrix into two components: $A^{(v)} = S \odot G^{(v)}$. Here, $S$ is a shared shape mask capturing the underlying structure (the "where" of connections), while $G^{(v)}$ is a model-specific magnitude matrix (the "how much" of the connection).
  3. Structural Pursuit (k-block diagonality): In an ideal scenario, an affinity matrix for $k$ objects should be $k$-block diagonal. The authors enforce this property on the shared mask $S$ using a soft regularizer based on the sum of the smallest eigenvalues of the Laplacian matrix.
  4. Optimization: A specialized alternative optimization scheme is used to solve this non-convex problem, with a mathematical proof provided for global convergence.

Experimental Evaluation

The method was evaluated on four diverse real-world benchmarks:

  • Hopkins155: The standard "clean" benchmark.
  • Hopkins12: A subset focusing on incomplete trajectories.
  • MTPV62: A complex dataset featuring occlusions and perspective effects.
  • KT3DMoSeg: Based on the KITTI dataset, featuring significant camera translation and complex backgrounds.

Key Results

The proposed method consistently outperformed state-of-the-art competitors across all benchmarks. On the challenging KT3DMoSeg dataset, the method achieved an average error rate of 4.58%, a significant improvement over the next best fusion-based competitor, "Subset," which recorded 8.08%.

Error rate comparison per sequence between the proposed method and the "Subset" baseline, demonstrating consistent performance gains across various clips.

Qualitatively, the model excels at distinguishing small objects from backgrounds and handling imbalanced clusters where one object has significantly more tracked points than another.

Qualitative comparison of motion segmentation results. "Ours" closely matches the Ground Truth (GT) across different scenes, effectively handling complex backgrounds and multiple moving objects compared to the "Subset" method.

Discussion and Future Work

By separating "structure" from "magnitude," the model can ignore high-magnitude but semantically incorrect affinities that often plague the Fundamental model. The $k$-block diagonal prior ensures that the final consensus matrix represents distinct, non-overlapping motions.

However, some limitations remain. The model shows sensitivity to the hyperparameter $\alpha_2$, which controls the strictness of the block-diagonal pursuit. Additionally, the fusion performance is bounded by the quality of the initial affinity matrices; if all input models fail to capture a specific motion, the fusion cannot recover it. Future work may also address the requirement of a fixed $k$, as the number of motions is often unknown in fully autonomous settings.

Summary updated

Abstract

Motion segmentation aims at separating motions of different moving objects in a video sequence. Facing the complicated real-world scenes, recent studies reveal that combining multiple geometric models would be a more effective way than just employing a single one. This motivates a new wave of model-fusion based motion segmentation methods. However, the vast majority of models of this kind merely seek consensus in spectral embeddings. We argue that a simple consensus might be insufficient to filter out the harmful information which is either unreliable or semantically unrelated to the segmentation task. Therefore, how to automatically select valuable patterns across multiple models should be regarded as a key challenge here. In this paper, we present a novel geometric-model-fusion framework for motion segmentation, which targets at constructing a consistent affinity matrix across all the geometric models. Specifically, it incorporates the structural information shared by affinity matrices to select those semantically consistent entries. Meanwhile, a multiplicative decomposition scheme is adopted to ensure structural consistency among multiple affinities. To solve this problem, an alternative optimization scheme is proposed, together with a proof of its global convergence. Experiments on four real-world benchmarks show the superiority of the proposed method.
BibTeX
@inproceedings{jiang2021what,
  author = {Yangbangyan Jiang and Qianqian Xu and Ke Ma and Zhiyong Yang and Xiaochun Cao and Qingming Huang},
  title = {{What to Select: Pursuing Consistent Motion Segmentation from Multiple Geometric Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i2.16264},
}
Figure