Introduction
Motion segmentation is a fundamental task in computer vision, particularly for autonomous systems that must distinguish between static background and multiple moving objects. While spectral clustering is a popular technique for grouping trajectories based on motion similarity, its practical application is often hindered by the requirement to predefine the number of clusters ($K$). In dynamic real-world environments with moving cameras and varying depths, determining $K$ manually is unfeasible. This paper proposes a unified model selection technique that automatically estimates the number of motion groups by ensembling multiple clustering validation indices.
Methodology
The core of the proposed approach is a Unified Model Selection framework that moves away from reliance on a single mathematical heuristic. Instead, it leverages the strengths of four established clustering metrics:
- Silhouette Score: Evaluates cluster cohesion and separation.
- Eigengap Heuristic: Identifies the number of clusters based on the largest gap between consecutive eigenvalues of the graph Laplacian.
- Davies-Bouldin Index: Measures the ratio of within-cluster scatter to between-cluster separation.
- Calinski-Harabasz Index: Assesses the ratio of between-cluster dispersion to within-cluster dispersion.
The Workflow
The system processes video data through a multi-stage pipeline. First, dense point trajectories are extracted from image frames. These trajectories are then used to generate motion affinity matrices based on different cues, such as epipolar geometry and optical flow.
To perform model selection, the affinity matrices ($A$) are converted into distance matrices ($D = 1 - A$). For a range of potential motion groups (typically 2 to 5), normalized confidence scores are calculated for each of the four metrics. The final selection is determined by Ensemble Averaging, where the number of motions with the highest average confidence across all criteria is selected. This method is more robust than simple consensus voting because it incorporates the relative strength of each metric's confidence.
Experimental Results
The method was evaluated on the KT3DMoSeg dataset, a challenging benchmark derived from KITTI that features 22 sequences with significant motion degeneracy and parallax.
Performance Metrics
The proposed "Average" ensemble method demonstrated superior robustness compared to individual metrics and simple voting schemes. As shown in the table below, the unified approach achieved high accuracy in predicting the number of clusters, particularly when using fused affinity matrices.
Key quantitative findings include:
- Prediction Accuracy: The method correctly identified the exact number of motion groups in 63.64% of cases using fused affinities.
- Mean Squared Error (MSE): The system achieved an MSE of 1.091 for cluster prediction, a significant improvement over random guessing (3.091).
- Segmentation Accuracy: The final motion segmentation error rate was 12.03%. While this is higher than the 5.78% error rate achieved when the ground truth $K$ is provided, it represents a competitive result for a fully automated system.
Conclusion and Future Work
By ensembling multiple validation indices, the proposed unified model selection technique mitigates the failure modes of individual heuristics. While the system still faces challenges in sequences with high motion degeneracy (particularly those with four motion groups), it significantly narrows the gap between automated and supervised motion segmentation. Future research will focus on expanding the search range for $K$ and improving the quality of input affinity matrices to handle more complex scene geometries.
Summary updated