A LiDAR scan of a walking person typically holds a few dozen to a few hundred points, all on the side facing the sensor. Over a two-second track, the same person shows the sensor many more points from many angles. For cars, the field already exploits this. TraPCC stacks a vehicle's points across frames using its tracked 3D bounding box, which moves as one rigid piece, and uses the stack as a free training target for shape completion. That trick fails for anything that bends. A pedestrian's legs swing between frames, so stacking points with one rigid transform per frame smears limbs into a blur. The static-scene habit of treating each frame as a fixed snapshot of an unchanging object becomes a source of corrupted labels.
We propose to replace the rigid stacking step with a learned deformation: a small motion field, constrained to bend as little as possible, that moves each frame's sparse points into a shared reference pose. The warped stack then serves two jobs. Projected into each camera frame, it gives denser depth supervision on moving actors for an image-guided depth completion network (a model that turns sparse LiDAR plus an RGB image into a per-pixel depth map). Warped back to each frame as a full 3D cloud, it gives self-supervised targets for completing the actor's unseen back side. In two sentences, the claim is this. Non-rigid accumulation turns the sensor's own temporal redundancy into clean training targets for articulated actors, where rigid accumulation produces smeared ones. Those targets improve both 2.5D depth completion and full 3D shape completion of people and other deformable objects without templates or manual labels.
Core Claim
The bottleneck for dynamic, deformable actors under sparse sensing is mainly supervision quality, not network architecture. Existing depth completion networks such as NLSPN, which spreads sparse depth to nearby pixels using RGB-learned affinities, and CompletionFormer, which adds global transformer attention to that propagation, treat each frame in isolation. They output only the visible surface. Point cloud completion networks such as PCN learn global shape priors from complete synthetic CAD models, which are rigid. TraPCC shows that real temporal aggregation can replace synthetic ground truth, but only under rigid motion. We claim that a template-free, as-rigid-as-possible deformation field fitted per track will make aggregation valid for articulated actors. We further claim that the gain will grow with how much the actor deforms.
Research Program
The pipeline has three parts. The first is the accumulator, which is the contribution. For each tracked actor, we take its LiDAR points $P_t$ in frame $t$, after coarse rigid alignment by the track's bounding box. We then fit a residual warp $W_\theta(x, t)$, a small multilayer perceptron with parameters $\theta$, that maps a point $x$ observed at time $t$ into a canonical reference pose. The fitting objective is
$$ \mathcal{L}(\theta)=\sum_{t}\sum_{t'\neq t}\mathrm{CD}_{\text{part}}\big(W_\theta(P_t,t),\,W_\theta(P_{t'},t')\big)+\lambda_f\sum_t \mathcal{L}_{\text{flow}}(\theta,t)+\lambda_a\,\mathcal{L}_{\text{ARAP}}(\theta). $$
Here $\mathrm{CD}_{\text{part}}$ is a one-sided, partial Chamfer distance, meaning each point is matched only to nearby points, so that surfaces seen in one frame but not another are not forced together. $\mathcal{L}_{\text{flow}}$ penalizes disagreement between the image-plane motion implied by the warp and dense optical flow from an off-the-shelf estimator. We reuse the flow-distillation idea from SplatFlow, which aligns Gaussian motion with SEA-RAFT flow. $\mathcal{L}_{\text{ARAP}}$ is the as-rigid-as-possible penalty on a nearest-neighbor graph of points, which allows bending but discourages stretching. This regularizer comes from 4DTAM, which pairs it with a neural warp field to track deforming surfaces.
4DTAM fits its warp to dense RGB-D frames and reconstructs only observed surfaces. We fit to LiDAR that is roughly two orders of magnitude sparser, and we use the result as a label generator rather than as the final reconstruction. Optical flow carries most of the motion constraint here. ARAP keeps the warp well-posed where LiDAR is too sparse to constrain it. Each canonical point receives a confidence score equal to the number of frames that agree on its position. Low-confidence points are dropped from the targets.
The second part is front-surface depth supervision. We warp the canonical cloud into each frame and project it into the camera. We keep only points that a z-buffer test marks as visible, and add them to the depth targets on actor pixels. We then retrain CompletionFormer unchanged, with this extra supervision.
The third part is back-surface completion. We train a completion network to map a single-frame partial actor cloud, optionally with RGB features sampled at its points, to the full canonical cloud warped into that frame. We use TraPCC's Modified Chamfer Distance loss. Some actors never turn, so no single track shows their back. The network must therefore learn a population-level prior from the many tracks that do show it. This is the same argument TraPCC relies on, now extended to people.
Baselines
The naive reference points come in two kinds. For 3D completion, the reference is the raw single-frame points with no completion. For depth completion, it is CompletionFormer trained with its standard targets. On the KITTI Depth Completion benchmark, CompletionFormer reports 708.87 mm RMSE (CompletionFormer) and NLSPN reports 741.68 mm (NLSPN). We retrain CompletionFormer ourselves with the same schedule, so the only difference between runs is the extra actor-pixel supervision. We report RMSE and MAE both overall and on a dynamic-actor pixel subset that we define from tracks. We found no reported result on Lacuna for depth completion error restricted to moving pedestrians, so we establish that number ourselves. Beating this baseline means a clear RMSE drop on actor pixels with no regression on background pixels. A small overall change is expected, because actors cover few pixels.
The strongest competitor for the core question is TraPCC-style rigid aggregation, run through our exact pipeline. It uses the same tracks, the same completion network, the same loss, the same epochs, and the same tuning budget. The only change is that the residual warp is fixed to identity, so frames are stacked with box transforms alone. TraPCC reports meanCD 0.0973 on KITTI vehicles (TraPCC). We found no reported result on Lacuna for its performance on pedestrians or cyclists. Beating it means lower Chamfer distance on articulated classes, with the gap widening as articulation increases, while staying within noise on vehicles. On vehicles the two methods should coincide.
For humans specifically, the strongest template-based competitor is fitting a parametric body model. VoteHMR votes for skeletal joints from partial point clouds and regresses a full SMPL body mesh. It reports 24.44 mm mean Chamfer distance on Berkeley MHAD and remains stable at 250 input points. We run VoteHMR on the same pedestrian crops and use its meshes as completion targets in place of our accumulated clouds.
Two ablations isolate the key ingredient. One removes the flow term, leaving LiDAR geometry and ARAP alone. The other removes ARAP, leaving an unregularized warp, which we expect to collapse into degenerate solutions that drive Chamfer distance to zero.
Why not just…?
Why not just run CompletionFormer, lift its depth to 3D, and feed PCN? This cascade answers neither failure mode. Depth completion still sees only the front surface, and it bleeds depth across object boundaries (NLSPN). EventDC further reports that RGB-guided completion degrades under motion blur and LiDAR–camera misalignment around moving objects. PCN's prior is learned on rigid CAD shapes. Litany et al. show that rigid-trained completion does poorly on articulated humans: 4.43 cm error for 3D-EPN versus 2.51 cm for their deformation-aware model on DFAUST scans. We include the cascade as a baseline, so the claim is measured rather than assumed.
Why not just aggregate with rigid boxes, as TraPCC does? This is the main threat. At long range a pedestrian may be only a handful of points, and limb motion may then sit below sensor noise. If so, rigid stacking would be good enough. The experiment settles the question by binning tracks by articulation magnitude. On synthetic data we measure this from ground-truth joint motion. On real data we measure it as the residual flow after the box transform. We then report the non-rigid minus rigid gap in each bin. If the gap is flat and near zero, the paper's claim is wrong, and we will say so.
Why not just fit SMPL with something like VoteHMR? For unclothed, standard human bodies this may match or beat us. That is why it is a baseline rather than a straw man. However, it produces only a body shell. WonderHuman, which hallucinates unseen human backs with diffusion priors on top of SMPL registration, documents failures on loose clothing and on non-SMPL deformation. SMPL also says nothing about cyclists, strollers, animals, or cloth. We therefore evaluate on clothed and non-human deformables, and we expect the template method to win on bare bodies and lose elsewhere.
Why not just mask dynamic actors out? Masking protects the static map but leaves the system blind to the actors that matter most (Robust Dense Mapping for Large-Scale Dynamic Environments). It also inherits segmentation drift, which Split4D identifies as a source of permanent 3D corruption. It is not a competitor on the completion metrics we care about.
Proposed Experiments
We need ground truth for back surfaces, which real LiDAR never provides, so the first test is controlled. We take the Sim4D benchmark, which contains 50 animated meshes of cloth, animals, and mechanical parts with ground-truth geometry. We render its depth through a simulated sparse-LiDAR pattern at densities from 5% down to under 1% of pixels. We measure Chamfer distance of the accumulated targets, and of the trained completion network, against the full meshes, comparing rigid, non-rigid, and ablated accumulators. We repeat the test on the CARLA-based EventDC-FullSyn split, which provides animated pedestrians in driving scenes with ground-truth trajectories. We also run DFAUST at matched sparsity, to compare with the 2.51 cm figure of Litany et al.
On real data, we build pedestrian and cyclist tracks from KITTI Tracking and Waymo. Without back-surface ground truth, our main real-data metric is held-out-frame consistency. For each track we withhold 20% of the frames from accumulation and measure how well targets warped into those frames predict the withheld LiDAR points. We also evaluate actor-pixel depth completion on KITTI Depth Completion, and report MHAD Chamfer distance against VoteHMR's 24.44 mm on real Kinect humans.
The decisive check is the articulation-binned gap between non-rigid and rigid accumulation, run on the same tracks with the same downstream networks. The result should be close to zero for vehicles and positive and growing for limbs and cloth. A second check concerns actors that never turn. If the completion network still recovers plausible backs on held-out tracks from Sim4D and CARLA, measured against ground truth, then the cross-track population prior is working. If it recovers them only for actors that turned, accumulation alone carries the result and the completion claim must be narrowed.
References
- Self-Supervised Point Cloud Completion on Real Traffic Scenes Via Scene-Concerned Bottom-Up Mechanism
- Non-local Spatial Propagation Network for Depth Completion
- CompletionFormer: Depth Completion with Convolutions and Vision Transformers
- PCN: Point Completion Network
- SplatFlow: Self-Supervised Dynamic Gaussian Splatting in Neural Motion Flow Field for Autonomous Driving
- 4DTAM: Non-Rigid Tracking and Mapping via Dynamic Surface Gaussians
- VoteHMR: Occlusion-Aware Voting Network for Robust 3D Human Mesh Recovery from Partial Point Clouds
- Event-Driven Dynamic Scene Depth Completion
- Deformable Shape Completion With Graph Convolutional Autoencoders
- WonderHuman: Hallucinating Unseen Parts in Dynamic 3D Human Reconstruction
- Robust Dense Mapping for Large-Scale Dynamic Environments
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation