Trajectory Regularization Enhances Self-Supervised Geometric Representation

Jiayun Wang, Stella X. Yu, Yubei Chen

TL;DR

Standard self-supervised learning (SSL) focuses on making features invariant to viewpoint, which helps classification but destroys geometric information. This paper introduces TrajSSL, which uses a novel trajectory regularization loss on mid-layer features to learn representations that are both semantically rich and pose-aware, significantly improving generalization to novel objects and viewpoints.

What Problem Does This Paper Address?

In computer vision, identifying what an object is (semantic categorization) is only half the battle. For applications like robotics or autonomous driving, understanding how an object is presented (its pose or viewpoint) is equally critical.

Current Self-Supervised Learning (SSL) methods, such as SimCLR or VICReg, are designed to be "view-invariant." They try to map different angles of the same car to the exact same feature vector. While great for classification, this "invariance" actively discards the geometric information needed for pose estimation. Furthermore, there has been a lack of standardized benchmarks to evaluate whether SSL models are learning these crucial geometric properties.

Methods

The authors propose two main strategies to create pose-aware representations:

  1. Viewpoint Trajectory Regularization ($L_{traj}$): Instead of just looking at pairs of images, the model is trained on unlabeled image triplets $\{X_L, X_C, X_R\}$ representing small, sequential viewpoint changes (a trajectory). The new loss function encourages the representations of these triplets to form a smooth, locally linear path on the feature hypersphere. By maximizing the cosine similarity between projected difference vectors, the model learns to organize its latent space according to physical pose changes.
The trajectory regularization loss enforces local linearity in the feature space for adjacent views.
  1. Mid-Layer Feature Utilization: The researchers discovered that the final layers of standard SSL models are the most pose-invariant (and thus the worst for pose estimation). By using intermediate layers (like res block 3 in a ResNet-18), they can tap into features that naturally retain more geometric detail. To handle the high dimensionality of these layers, they introduce a simple compression head.
The TrajSSL architecture combines a standard semantic loss with the new trajectory regularization loss.

Datasets

  • ShapeNet-based Benchmark: A new object-centric dataset featuring 1.5M images. It includes 13 in-domain and 11 out-of-domain semantic categories, with poses sampled uniformly using Fibonacci lattices to ensure even coverage.
  • Carvana: A real-world dataset of rotating cars used to test how well models trained on synthetic data generalize to real photos.
  • Objaverse: A large-scale, diverse 3D dataset used to further validate the universality of the approach.

Key Insights and Ideas

  • Invariance is a Double-Edged Sword: The very objective that makes SSL successful for classification (view-invariance) makes it fail at geometry. A "pose-aware" model needs to balance these competing goals.
  • Geometry Lives in the Middle: Geometric tasks are mid-level vision tasks. The paper shows that mid-layer features outperform final-layer features by 10-20% in pose estimation tasks.
  • Emergent Global Pose Awareness: By regularizing local pose changes (the triplets), the model develops an emergent global understanding of pose. It learns to organize objects in the feature space such that similar poses are clustered together, even for object categories it has never seen before.
The learned representation ideally disentangles object identity from object pose.

Key Results

  • Pose Estimation Gains: Adding the trajectory loss ($L_{traj}$) consistently improved relative pose accuracy by about 4% across different SSL baselines without harming semantic classification accuracy (which stayed around 85-86%).
  • Out-of-Domain Generalization: TrajSSL proved remarkably robust, often matching or even outperforming supervised methods when tested on unseen object categories or novel viewpoints.
  • Real-World Transfer: Models trained purely on synthetic ShapeNet data successfully transferred to the real-world Carvana dataset, achieving over 91% pose accuracy.

Limitations and Future Work

  • Synthetic Data Reliance: The primary benchmark is built on rendered 3D models. While the Carvana results are promising, more extensive testing on diverse real-world video data is needed.
  • Scope of Geometric Tasks: The paper focuses on 3D pose estimation (azimuth and elevation). Future work could extend this to more complex tasks like 6-DoF pose estimation or depth map prediction.
  • Data Acquisition: The method requires image triplets from trajectories. While natural in video or robotic streams, this data is less common in static image web-crawls compared to the single images used in standard SSL.

Summary updated

Abstract

Learning visual features from unlabeled images has proven successful for semantic categorization, often by mapping different $views$ of the same object to the same feature to achieve recognition invariance. However, visual recognition involves not only identifying $what$ an object is but also understanding $how$ it is presented. For example, seeing a car from the side versus head-on is crucial for deciding whether to stay put or jump out of the way. While unsupervised feature learning for downstream viewpoint reasoning is important, it remains under-explored, partly due to the lack of a standardized evaluation method and benchmarks. We introduce a new dataset of adjacent image triplets obtained from a viewpoint trajectory, without any semantic or pose labels. We benchmark both semantic classification and pose estimation accuracies on the same visual feature. Additionally, we propose a viewpoint trajectory regularization loss for learning features from unlabeled image triplets. Our experiments demonstrate that this approach helps develop a visual representation that encodes object identity and organizes objects by their poses, retaining semantic classification accuracy while achieving emergent global pose awareness and better generalization to novel objects. Our dataset and code are available at http://pwang.pw/trajSSL/.
BibTeX
@inproceedings{wang2024trajectory,
  author = {Jiayun Wang and Stella X. Yu and Yubei Chen},
  title = {{Trajectory Regularization Enhances Self-Supervised Geometric Representation}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=dUxn8kOssH},
}
Figure