Moving Objects as Free Multi-View Capture: Learning Object SLAM Shape Priors Without CAD Models

Object-level SLAM systems estimate the camera's path together with a map of whole objects (cars, chairs) instead of loose feature points, and they work best when they carry a learned "shape prior." A shape prior is a model that knows what cars generally look like, so it can fill in the far side of a car the camera has never seen. Today those priors are trained on clean synthetic CAD meshes, and the systems that use them assume nothing in the world moves. This proposal inverts both assumptions. A car that drives past, turns, or overtakes a camera shows the camera its front, side, and rear within seconds. If the SLAM system already tracks that car as a rigid object with its own pose over time, then every moving car becomes a small object-centric multi-view capture, gathered for free. The idea is to use these captures as the training data for the shape prior. The dynamic objects that break static reconstruction become the source of 3D supervision that CAD datasets currently provide.

Concretely, we would build a stereo driving SLAM system that treats each vehicle as a node in a joint optimization graph, whether it is parked or moving. The graph produces object-frame-aligned depth and mask observations. These train a category shape model with no 3D labels, and the model is then plugged back into the graph to constrain partly visible and occluded objects in new sequences. The testable claim is that a prior trained this way, from a fleet's own unlabelled logs, matches or beats a ShapeNet-trained prior for object pose and shape on real driving data. We also claim that motion is what makes this work: the same pipeline restricted to static objects should learn a worse prior, because it sees fewer viewpoints per instance.

Core Claim

Two strands in the file point at the same missing piece from opposite sides. DSP-SLAM jointly optimizes camera poses, map points, 6-DoF object poses and 64-dimensional DeepSDF shape codes. DeepSDF is a network that maps a latent code and a 3D point to a signed distance to the surface. NodeSLAM does something similar with a 16-dimensional voxel VAE. Both need complete 3D CAD shapes for training, and both treat every object as a static landmark. The dynamic multi-body systems do track moving objects in the graph. These include CubeSLAM, the stereo tracker of Li et al., CTO-SLAM and Multi-object Monocular SLAM. But they represent those objects only as cuboids, contour points or keypoint wireframes. The CTO-SLAM authors themselves note that camera accuracy stays hostage to dynamic-object tracking errors.

Our claim is that the object-motion variables these dynamic systems already estimate are exactly what is needed to learn a DSP-SLAM-style prior without CAD data. Once an object's rigid trajectory is known, all its stereo depth and mask observations can be expressed in one object frame. The direction of travel also gives a canonical forward axis for free. This matters because deciding which way is "front" is the hard alignment problem that Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos solves with DINOv2 features and a 3D cycle-consistency filter. That paper, however, needs static objects filmed by a walking camera and offline structure-from-motion. Here the object moves, the camera moves, and the separation of the two motions is done by the pose graph.

Research Program

The system runs in rounds over a corpus of unlabelled stereo driving logs. In round zero, vehicles enter the graph as cuboid nodes, in the style of CubeSLAM. They are initialized from 2D detections and linked across time by a bicycle motion factor, a simple car-kinematics model of the kind Li et al. use. A per-object mini-bundle adjustment refines their trajectories. From the resulting tracks we select those with low reprojection residual and wide angular coverage in the object frame. We transform their stereo depth and instance masks into that frame, with heading taken from the object's velocity; for parked cars, heading comes from the cuboid. These observations train an auto-decoder signed distance model: a shared decoder plus one latent code per instance. The loss combines depth-rendering and silhouette-rendering terms with a Gaussian prior on the codes.

Using stereo depth rather than silhouettes alone is deliberate. Multi-View Consistency as Supervisory Signal shows that mask-only supervision gets surprisingly close to 3D supervision: 0.53 versus 0.62 mean IoU on ShapeNet chairs. It is also capped by the visual hull, meaning it cannot recover concave regions that no silhouette reveals.

In round one, the cuboid factor is replaced by DSP-SLAM-style factors for every vehicle, parked or moving, with each object's pose now a function of time. Writing $T_{c,t}$ for the camera pose at time $t$, $T_{o,t}$ for object $o$'s pose, $z_o$ for its shape code, and $f_\theta$ for the learned decoder, the object terms of the energy are

$$E = \sum_{o,t} \Big( \sum_{p \in P_{o,t}} \rho\big(f_\theta(T_{o,t}^{-1} T_{c,t}\, p;\, z_o)\big) + \lambda_s\, E_{\text{sil}}(T_{c,t}, T_{o,t}, z_o) \Big) + \sum_{o,t} E_{\text{motion}}(T_{o,t}, T_{o,t+1}) + \lambda_z \sum_o \lVert z_o \rVert^2.$$

Here $P_{o,t}$ is the set of stereo points on object $o$ at time $t$, back-projected from camera coordinates; $\rho$ is a robust loss; $E_{\text{sil}}$ compares the rendered silhouette with the detected mask; and $E_{\text{motion}}$ is the bicycle-model factor. The weights $\lambda_s$ and $\lambda_z$ balance the silhouette and code-regularization terms. The static-background terms are standard.

Refined trajectories feed the next training round. We then study whether the rounds improve or drift, since errors in poses could blur shapes, which could in turn bias poses. We guard against this by filtering tracks on residual and by holding out sequences that never enter training. A secondary analysis asks how many distinct observed viewpoints per instance the prior needs, and whether moving objects supply them more cheaply than static ones.

Baselines

The naive reference point is static stereo SLAM with dynamic objects masked out by an off-the-shelf segmentor. On camera pose this is represented by ORB-SLAM2, which Li et al. report drifting up to 19.35 m on dynamic KITTI sequences. It has no object output at all. We run it on the identical KITTI Tracking sequences and stereo input.

The strongest camera-only competitor is D^3FlowSLAM, a self-supervised system that separates ego-motion flow from object flow and masks moving pixels out of differentiable bundle adjustment. It reports 2.852 m ATE on KITTI, though that is a monocular figure on its own sequence set, so we rerun its released model on our splits rather than compare against the number. Beating it means lower camera ATE on sequences with many moving vehicles, stratified by the fraction of the image covered by vehicles.

The strongest object-level competitor is our own graph with a ShapeNet-trained DeepSDF car prior swapped in. This is DSP-SLAM's prior extended with time-varying poses. It uses the same solver, detections, iteration budget and code dimension, and it isolates the question of CAD versus self-captured supervision. The published dynamic references are the stereo tracker of Li et al., which reports about 1.03 m error, and CTO-SLAM. CTO-SLAM reports only relative gains of 20% RPE and 33% dynamic-object ATE over DynaSLAM II, so no absolute value is found in the file; we rerun it where code allows. Multi-object Monocular SLAM reports roughly 3 m ATE on sequences over 100 m, but it is monocular and serves only as context.

A cuboid-only version of our graph (round zero, never upgraded) isolates the value of any dense prior. It is the configuration that CubeSLAM's 1.78% KITTI odometry translation error represents. A per-instance fusion baseline accumulates each tracked object's depth into its own implicit surface with no shared prior. It isolates whether the learned prior matters, or whether the tracks alone suffice.

The key ablation trains the prior only on parked vehicles. The data budget is matched: the same number of instances and frames, differing only in whether the objects moved. This isolates the central claim that object motion is what makes the capture informative. Tuning budgets are equalized by giving every learned variant the same hyperparameter search on a shared validation split.

Why not just…?

Why not just use the ShapeNet car prior that DSP-SLAM already ships? For cars this is the most serious threat, because CAD cars are plentiful. It may simply win on shape accuracy. Our experiments report the head-to-head directly, and we would count parity as a meaningful result: it would mean the CAD dependency can be removed for free. The case for self-capture rests on two things this comparison measures. The first is the sim-to-real gap on real vehicles. Virtual KITTI 2 documents that synthetic-to-real gaps persist even when they narrow. The second is categories ShapeNet covers poorly on KITTI, namely vans, trucks and trams, which we report separately. If the CAD prior wins on every category, the proposal's contribution shrinks to the scalability argument, and we would say so.

Why not just use cuboids, as CubeSLAM does? On camera trajectory alone, cuboids may be nearly as good, since the object mainly acts as a rigid anchor. We therefore do not rest the claim on camera ATE. The prediction is that dense priors help object pose most when a vehicle is truncated at the image border or partly occluded. In those cases a cuboid fitted to a partial mask shifts its center, while a shape model fitted to visible surface does not. We therefore stratify object pose error by visible fraction. If cuboids match the dense prior even in the heavily occluded bins, that is a negative result on dense priors for dynamic SLAM, and we would report it as such.

Why not just mask moving objects out and keep them out of the graph? Masking discards the constraints that Li et al. show reduce drift from 19.35 m to about 1.03 m on dynamic KITTI sequences. It also produces no object trajectories or shapes, which downstream planning needs. D^3FlowSLAM is included precisely because it is the best version of this alternative. It may win on camera ATE in sparse traffic, and our traffic-density stratification is designed to show where that crossover happens.

Proposed Experiments

Training logs come from KITTI raw stereo sequences disjoint from the KITTI Tracking evaluation sequences. We also use Virtual KITTI 2, whose dense depth and instance masks allow exact evaluation.

Object pose is scored against KITTI Tracking 3D boxes by per-object trajectory ATE and by heading error. Camera pose is scored by ATE and RPE. Shape completion is scored without CAD ground truth by predicting the depth of held-out frames in which an object shows a side unseen during fitting. On KITTI the reference for these held-out views is LiDAR; on Virtual KITTI 2 it is the dense rendered depth, where we also compute Chamfer distance against the synthetic meshes.

The first decisive check is the moving-versus-parked ablation at matched data. If motion-derived training is not clearly better on held-out-view depth error, the core claim fails. The second is the comparison with the CAD prior on real KITTI, broken down by vehicle category. The third is occlusion-stratified object pose error against the cuboid-only graph, which determines whether dense priors earn their cost in dynamic scenes. Finally, we track all metrics across training rounds on held-out sequences to show whether the self-training loop converges, drifts, or collapses toward blob-like shapes.