Learned Trajectory Embedding for Subspace Clustering

Yaroslava Lochman, Carl Olsson, Christopher ZachCVPR 2024 · DOI

Introduction

Motion segmentation is a fundamental task in computer vision that involves grouping tracked points in a video sequence based on their underlying motion models. This task is inherently challenging due to the "chicken-and-egg" nature of the problem: accurate grouping requires knowledge of the motion models (subspaces), while estimating these models requires correctly grouped points. Traditional methods often struggle with high-dimensional motion, missing data from occlusions, and the computational inefficiency of robust estimators like RANSAC when dealing with multiple moving objects.

Proposed Methodology

The authors introduce a novel deep learning framework that decouples feature extraction from geometric motion estimation. The architecture consists of three primary components:

  1. Feature Extractor: Utilizing a PointNet-inspired encoder with 1D temporal convolutions, the system maps input trajectories of arbitrary lengths into a fixed 128-dimensional embedding space.
  2. Metric Learning: The network is trained using a contrastive InfoNCE loss ($L_{InfoNCE}$), which encourages trajectories from the same motion to cluster together while pushing distinct motions apart in the embedding space.
  3. Subspace Estimator: To ground the embeddings in physical geometry, a coordinate-MLP "decodes" the features to reconstruct trajectory basis functions. The model employs damped cosine basis functions to represent temporal motion. A geometric residual loss ($L_{Residual}$) ensures that the learned features are consistent with the underlying linear subspaces.
The proposed architecture: trajectories are mapped to embeddings, refined via a geometric subspace estimator, and clustered for final segmentation.

For trajectories with missing data, the model employs an iterative completion process. It estimates the subspace from visible segments, predicts missing coordinates, and refines the estimate until convergence.

Key Insights: The Blessing of Dimensionality

A core contribution of this work is the observation that while high-dimensional models are typically harder to fit, longer trajectories actually make the underlying motion more unique and identifiable. This "blessing of dimensionality" allows simple clustering in a learned feature space to be highly effective.

Affinity matrices for varying trajectory lengths (F). As the number of frames increases, the block-diagonal structure becomes more distinct, facilitating easier clustering.

Furthermore, the authors validate the "Disjoint Subspace Assumption," showing that in real-world datasets, different motions rarely share the same subspace, justifying the use of discriminative embeddings.

Experimental Evaluation

The model was rigorously tested on three major benchmarks:

  • Hopkins155: Achieved a state-of-the-art classification error of 0.62%, significantly outperforming Sparse Subspace Clustering (SSC) at 2.45%.
  • KT3DMoSeg: In real-world driving scenarios with heavy occlusions (up to 75%), the model maintained a median error of 0.80%, whereas traditional methods often fail completely.
  • Hopkins12: Demonstrated robust trajectory completion capabilities on sequences with up to 23% missing data.

Beyond accuracy, the method is exceptionally efficient. The entire clustering pipeline runs in approximately 7-10 milliseconds, making it suitable for real-time applications and orders of magnitude faster than iterative statistical alternatives.

Conclusion and Future Work

This research demonstrates that mapping trajectories to a discriminative, geometrically-informed embedding space provides a powerful solution for motion segmentation. While the current model assumes linear subspaces—an approximation of the affine camera model—future work aims to extend this to non-rigid deformations and incorporate global context to further improve performance on highly noisy or short trajectories.

Summary updated

BibTeX
@inproceedings{lochman2024learned,
  author = {Yaroslava Lochman and Carl Olsson and Christopher Zach},
  title = {{Learned Trajectory Embedding for Subspace Clustering}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01806},
}
Figure