Abstract
Motion segmentation is a fundamental task in computer vision, yet real-world complexities like perspective effects and occlusions often render single geometric models insufficient. This paper introduces a novel geometric-model-fusion framework that moves beyond simple consensus-seeking. By employing a multiplicative decomposition strategy and enforcing $k$-block diagonal structural consistency across multiple geometric models (affine, homography, and fundamental), the proposed method successfully filters out unreliable affinities to produce highly accurate, unified segmentations of moving objects in complex video scenes.
The Challenge of Multi-Model Fusion
Motion segmentation involves grouping feature point trajectories in a video into distinct moving objects. However, the diversity of real-world scenes—ranging from tiny objects to significant perspective shifts—makes it nearly impossible for a single geometric model, such as the standard affine model, to capture all motions accurately.
While recent research has shifted toward "model fusion" to combine the strengths of multiple models, most existing methods only seek consensus late in the pipeline, typically within the spectral embedding space. This delay often fails to filter out "harmful" information, such as false-positive connections between different objects present in raw affinity matrices. The core challenge lies in automatically selecting valuable patterns across multiple models while discarding noise.
Methodology: Structural Pursuit via Multiplicative Decomposition
The proposed framework pursues Consistent Motion Segmentation with Structural Pursuit through a multi-step process:
- Multi-Model Affinity Construction: Multiple affinity matrices ($A^{(v)}$) are generated using three classic geometric models: Affine, Homography, and Fundamental. These are constructed using Sampson distance and the Ordered Residual Kernel (ORK).
- Multiplicative Decomposition: The central innovation is the decomposition of each model's affinity matrix into two components: $A^{(v)} = S \odot G^{(v)}$. Here, $S$ is a shared shape mask capturing the underlying structure (the "where" of connections), while $G^{(v)}$ is a model-specific magnitude matrix (the "how much" of the connection).
- Structural Pursuit (k-block diagonality): In an ideal scenario, an affinity matrix for $k$ objects should be $k$-block diagonal. The authors enforce this property on the shared mask $S$ using a soft regularizer based on the sum of the smallest eigenvalues of the Laplacian matrix.
- Optimization: A specialized alternative optimization scheme is used to solve this non-convex problem, with a mathematical proof provided for global convergence.
Experimental Evaluation
The method was evaluated on four diverse real-world benchmarks:
- Hopkins155: The standard "clean" benchmark.
- Hopkins12: A subset focusing on incomplete trajectories.
- MTPV62: A complex dataset featuring occlusions and perspective effects.
- KT3DMoSeg: Based on the KITTI dataset, featuring significant camera translation and complex backgrounds.
Key Results
The proposed method consistently outperformed state-of-the-art competitors across all benchmarks. On the challenging KT3DMoSeg dataset, the method achieved an average error rate of 4.58%, a significant improvement over the next best fusion-based competitor, "Subset," which recorded 8.08%.
Qualitatively, the model excels at distinguishing small objects from backgrounds and handling imbalanced clusters where one object has significantly more tracked points than another.
Discussion and Future Work
By separating "structure" from "magnitude," the model can ignore high-magnitude but semantically incorrect affinities that often plague the Fundamental model. The $k$-block diagonal prior ensures that the final consensus matrix represents distinct, non-overlapping motions.
However, some limitations remain. The model shows sensitivity to the hyperparameter $\alpha_2$, which controls the strictness of the block-diagonal pursuit. Additionally, the fusion performance is bounded by the quality of the initial affinity matrices; if all input models fail to capture a specific motion, the fusion cannot recover it. Future work may also address the requirement of a fixed $k$, as the number of motions is often unknown in fully autonomous settings.
Summary updated