A Unified Model Selection Technique for Spectral Clustering Based Motion Segmentation

Yuxiang Huang, John S. ZelekarXiv 2024 · DOI · arXiv

Introduction

Motion segmentation is a fundamental task in computer vision, particularly for autonomous systems that must distinguish between static background and multiple moving objects. While spectral clustering is a popular technique for grouping trajectories based on motion similarity, its practical application is often hindered by the requirement to predefine the number of clusters ($K$). In dynamic real-world environments with moving cameras and varying depths, determining $K$ manually is unfeasible. This paper proposes a unified model selection technique that automatically estimates the number of motion groups by ensembling multiple clustering validation indices.

Methodology

The core of the proposed approach is a Unified Model Selection framework that moves away from reliance on a single mathematical heuristic. Instead, it leverages the strengths of four established clustering metrics:

  1. Silhouette Score: Evaluates cluster cohesion and separation.
  2. Eigengap Heuristic: Identifies the number of clusters based on the largest gap between consecutive eigenvalues of the graph Laplacian.
  3. Davies-Bouldin Index: Measures the ratio of within-cluster scatter to between-cluster separation.
  4. Calinski-Harabasz Index: Assesses the ratio of between-cluster dispersion to within-cluster dispersion.

The Workflow

The system processes video data through a multi-stage pipeline. First, dense point trajectories are extracted from image frames. These trajectories are then used to generate motion affinity matrices based on different cues, such as epipolar geometry and optical flow.

The proposed motion segmentation pipeline, integrating per-frame tracking, optical flow, and ensemble-based spectral clustering.

To perform model selection, the affinity matrices ($A$) are converted into distance matrices ($D = 1 - A$). For a range of potential motion groups (typically 2 to 5), normalized confidence scores are calculated for each of the four metrics. The final selection is determined by Ensemble Averaging, where the number of motions with the highest average confidence across all criteria is selected. This method is more robust than simple consensus voting because it incorporates the relative strength of each metric's confidence.

Experimental Results

The method was evaluated on the KT3DMoSeg dataset, a challenging benchmark derived from KITTI that features 22 sequences with significant motion degeneracy and parallax.

Performance Metrics

The proposed "Average" ensemble method demonstrated superior robustness compared to individual metrics and simple voting schemes. As shown in the table below, the unified approach achieved high accuracy in predicting the number of clusters, particularly when using fused affinity matrices.

Comparison of model selection methods across different affinity types. The 'Average' ensemble method achieves the best overall performance.

Key quantitative findings include:

  • Prediction Accuracy: The method correctly identified the exact number of motion groups in 63.64% of cases using fused affinities.
  • Mean Squared Error (MSE): The system achieved an MSE of 1.091 for cluster prediction, a significant improvement over random guessing (3.091).
  • Segmentation Accuracy: The final motion segmentation error rate was 12.03%. While this is higher than the 5.78% error rate achieved when the ground truth $K$ is provided, it represents a competitive result for a fully automated system.

Conclusion and Future Work

By ensembling multiple validation indices, the proposed unified model selection technique mitigates the failure modes of individual heuristics. While the system still faces challenges in sequences with high motion degeneracy (particularly those with four motion groups), it significantly narrows the gap between automated and supervised motion segmentation. Future research will focus on expanding the search range for $K$ and improving the quality of input affinity matrices to handle more complex scene geometries.

Summary updated

BibTeX
@misc{huang2024unified,
  author = {Yuxiang Huang and John S. Zelek},
  title = {{A Unified Model Selection Technique for Spectral Clustering Based Motion Segmentation}},
  year = {2024},
  eprint = {2403.01606},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.01606},
}
Figure