Abstract
Inferring the pose and shape of vehicles in 3D from a movable platform still remains a challenging task due to the projective sensing principle of cameras, difficult surface properties e.g. reflections or transparency, and illumination changes between images. In this paper, we propose to use 3D shape and motion priors to regularize the estimation of the trajectory and the shape of vehicles in sequences of stereo images. We represent shapes by 3D signed distance functions and embed them in a low-dimensional manifold. Our optimization method allows for imposing a common shape across all image observations along an object track. We employ a motion model to regularize the trajectory to plausible object motions. We evaluate our method on the KITTI dataset and show state-of-the-art results in terms of shape reconstruction and pose estimation accuracy.
Introduction
Reconstructing vehicles in 3D from a moving camera is a classic "ill-posed" problem in computer vision. While LiDAR provides precise depth, cameras are cheaper and more common but face significant hurdles. Distant objects suffer from low resolution, and the shiny or transparent surfaces of vehicles often confuse standard stereo-matching algorithms. Furthermore, estimating shape frame-by-frame often leads to temporal inconsistency, where a vehicle appears to jitter or change size as it moves.
SAMP (Shape and Motion Priors) addresses these gaps by moving from 3D (single frame) to 4D (sequence) reconstruction. By enforcing the constraint that a single physical object should maintain a consistent shape and follow a plausible path over time, the framework can recover accurate models even from noisy or incomplete stereo data.
Methodology
The SAMP framework operates on a tracking-by-detection pipeline enhanced with a sophisticated optimization layer. The core technical contribution is the use of a learned low-dimensional "shape manifold."
Shape Manifold and SDFs
Instead of reconstructing meshes from scratch, the authors learned a manifold of car shapes. They converted various CAD models into Signed Distance Functions (SDFs)—a representation where every point in space stores its distance to the nearest surface. Using Principal Component Analysis (PCA), they identified the primary ways car shapes vary, allowing any vehicle to be represented by a low-dimensional vector $z$.
Joint Optimization
The optimization process balances three primary components:
- Data Term: Alignment between the estimated shape and observed stereo depth points.
- Motion Term: Regularization of the vehicle's path to follow realistic trajectories (straight lines or circular arcs).
- Shape Prior: Ensuring the estimated shape remains "car-like" within the learned PCA manifold.
Crucially, the system enforces a Shared Shape Constraint, meaning the same shape parameters must explain observations across every frame in a track. This allows high-fidelity observations from close range to "fix" noisy reconstructions of the same vehicle when it was further away.
Results and Evaluation
The method was evaluated on the KITTI dataset, specifically the Stereo 2015 multi-view extension, which provides ground truth LiDAR points and pose annotations for 200 urban video sequences.
Key Findings
- Shape Accuracy: SAMP achieved an F1 score of 77.11% on the KITTI benchmark, significantly outperforming raw stereo baselines like libELAS (71.65%).
- Pose Estimation: The method reduced errors in both rotation and translation. Notably, the Median Absolute Error (MAE) for rotation remained stable even as the distance to the camera increased, whereas baseline methods saw errors spike at longer ranges.
- Model Completion: By fitting a complete 3D prior, the system can "fill in" parts of the car never seen by the camera, such as the undercarriage or the far side of the vehicle.
Conclusion
SAMP demonstrates that integrating temporal consistency and learned shape priors can overcome the inherent limitations of stereo-based 3D reconstruction. While currently focused on the car class, the framework provides a robust foundation for 4D scene understanding. Future work aims to expand the manifold to include more object classes like trucks and pedestrians, and to couple shape priors more tightly with the initial tracking process.
Summary updated