SAMP: Shape and Motion Priors for 4D Vehicle Reconstruction

Francis Engelmann, Jörg Stückler, Bastian Leibe

Abstract

Inferring the pose and shape of vehicles in 3D from a movable platform still remains a challenging task due to the projective sensing principle of cameras, difficult surface properties e.g. reflections or transparency, and illumination changes between images. In this paper, we propose to use 3D shape and motion priors to regularize the estimation of the trajectory and the shape of vehicles in sequences of stereo images. We represent shapes by 3D signed distance functions and embed them in a low-dimensional manifold. Our optimization method allows for imposing a common shape across all image observations along an object track. We employ a motion model to regularize the trajectory to plausible object motions. We evaluate our method on the KITTI dataset and show state-of-the-art results in terms of shape reconstruction and pose estimation accuracy.

Introduction

Reconstructing vehicles in 3D from a moving camera is a classic "ill-posed" problem in computer vision. While LiDAR provides precise depth, cameras are cheaper and more common but face significant hurdles. Distant objects suffer from low resolution, and the shiny or transparent surfaces of vehicles often confuse standard stereo-matching algorithms. Furthermore, estimating shape frame-by-frame often leads to temporal inconsistency, where a vehicle appears to jitter or change size as it moves.

SAMP (Shape and Motion Priors) addresses these gaps by moving from 3D (single frame) to 4D (sequence) reconstruction. By enforcing the constraint that a single physical object should maintain a consistent shape and follow a plausible path over time, the framework can recover accurate models even from noisy or incomplete stereo data.

Methodology

The SAMP framework operates on a tracking-by-detection pipeline enhanced with a sophisticated optimization layer. The core technical contribution is the use of a learned low-dimensional "shape manifold."

Shape Manifold and SDFs

Instead of reconstructing meshes from scratch, the authors learned a manifold of car shapes. They converted various CAD models into Signed Distance Functions (SDFs)—a representation where every point in space stores its distance to the nearest surface. Using Principal Component Analysis (PCA), they identified the primary ways car shapes vary, allowing any vehicle to be represented by a low-dimensional vector $z$.

Learning a shape manifold from CAD models using Signed Distance Functions (SDFs) and PCA.

Joint Optimization

The optimization process balances three primary components:

  1. Data Term: Alignment between the estimated shape and observed stereo depth points.
  2. Motion Term: Regularization of the vehicle's path to follow realistic trajectories (straight lines or circular arcs).
  3. Shape Prior: Ensuring the estimated shape remains "car-like" within the learned PCA manifold.

Crucially, the system enforces a Shared Shape Constraint, meaning the same shape parameters must explain observations across every frame in a track. This allows high-fidelity observations from close range to "fix" noisy reconstructions of the same vehicle when it was further away.

Results and Evaluation

The method was evaluated on the KITTI dataset, specifically the Stereo 2015 multi-view extension, which provides ground truth LiDAR points and pose annotations for 200 urban video sequences.

Qualitative comparison showing the noisy input stereo data versus the refined 4D output reconstruction.

Key Findings

  • Shape Accuracy: SAMP achieved an F1 score of 77.11% on the KITTI benchmark, significantly outperforming raw stereo baselines like libELAS (71.65%).
  • Pose Estimation: The method reduced errors in both rotation and translation. Notably, the Median Absolute Error (MAE) for rotation remained stable even as the distance to the camera increased, whereas baseline methods saw errors spike at longer ranges.
  • Model Completion: By fitting a complete 3D prior, the system can "fill in" parts of the car never seen by the camera, such as the undercarriage or the far side of the vehicle.

Conclusion

SAMP demonstrates that integrating temporal consistency and learned shape priors can overcome the inherent limitations of stereo-based 3D reconstruction. While currently focused on the car class, the framework provides a robust foundation for 4D scene understanding. Future work aims to expand the manifold to include more object classes like trucks and pedestrians, and to couple shape priors more tightly with the initial tracking process.

Summary updated

Abstract

Inferring the pose and shape of vehicles in 3D from a movable platform still remains a challenging task due to the projective sensing principle of cameras, difficult surface properties e.g. reflections or transparency, and illumination changes between images. In this paper, we propose to use 3D shape and motion priors to regularize the estimation of the trajectory and the shape of vehicles in sequences of stereo images. We represent shapes by 3D signed distance functions and embed them in a low-dimensional manifold. Our optimization method allows for imposing a common shape across all image observations along an object track. We employ a motion model to regularize the trajectory to plausible object motions. We evaluate our method on the KITTI dataset and show state-of-the-art results in terms of shape reconstruction and pose estimation accuracy.
BibTeX
@inproceedings{engelmann2017samp,
  author = {Francis Engelmann and Jörg Stückler and Bastian Leibe},
  title = {{SAMP: Shape and Motion Priors for 4D Vehicle Reconstruction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2017},
  url = {https://openreview.net/forum?id=bNGw6cN6piS},
}
Figure