Motion as a Free Multi-View Rig: Temporally Synchronized Local Frames for Completing Moving Objects

A depth sensor or LiDAR sees only the side of an object that faces it, so every single scan is a hollow shell. When the object moves (a car turning at an intersection, a box tumbling on a conveyor, a person rotating in place), the sensor sees a different side in each frame. In principle the sequence contains far more of the surface than any one frame. Two things stop current methods from using it. Shape-completion networks that work under arbitrary orientation, such as ESCAPE and REVNET, treat each frame on its own. Sequence models such as CaSPR require every frame to be handed to them already aligned into a canonical box. To fuse frames yourself you need the object's rotation between frames, and on partial scans that is exactly where the standard tool fails. A local reference frame is a small coordinate system (a local "compass") attached to each surface patch. These frames flip axes and collapse at the torn edges of partial scans.

The proposal is to make the motion itself repair the compasses. If two frames see the same surface patch, the patch's local frames at the two times must differ by the object's true rotation. If we go around a loop of frames, the rotations must compose back to the identity. We train a local-frame predictor with this loop-consistency signal and no ground-truth poses. The resulting frames are stable enough to register partial frames of a moving object. A rotation-invariant completion network then runs on the fused observation instead of a single shell. Two sentences for a colleague: moving objects show their sensor many sides over time, but fusing those views needs per-patch orientation estimates that are notoriously unstable on partial scans. We use the requirement that orientation changes compose consistently around loops of frames to train stable local frames without labels, then complete the object from all frames at once.

Core Claim

Temporal cycle-consistency is a supervisory signal for local reference frames that static methods do not have. It specifically fixes the failure modes listed in this literature: axis flips from asymmetric construction (LocoTrans), in-plane ambiguity on flat or symmetric patches (Zhang et al.), and collapse at partial-scan boundaries (Attaiki et al.).

Our concrete prediction is about when fusion helps. Completion error on a moving rigid object should fall with the angular range of the object's rotation over the sequence, and should be flat for pure translation, where no new sides are revealed. Per-frame equivariant completion cannot show this dependence at all. If the dependence does not appear, the idea is wrong.

Let $F_t^i \in \mathrm{SO}(3)$ be the predicted local frame of patch $i$ at frame $t$. For a patch matched across frames $s$ and $t$, the implied relative object rotation is $\hat R_{st}^i = F_t^i (F_s^i)^\top$. We combine these per-patch hypotheses into one robust estimate $\hat R_{st}$ by taking the consensus and rejecting outliers. We then train the frame predictor so that rotations compose around frame triplets:

$$ \mathcal{L}_{\text{cycle}} = \sum_{(s,t,u)} \big\| \hat R_{tu}\hat R_{st} - \hat R_{su} \big\|_F^2 + \lambda \sum_{i,(s,t)} w_{st}^i \big\| \hat R_{st}^i - \hat R_{st} \big\|_F^2 , $$

The first term asks that going from $s$ to $t$ to $u$ equal going directly from $s$ to $u$. The second asks each patch to agree with the consensus. The weight $w_{st}^i$ discounts patches the predictor marks as degenerate, and $\lambda$ balances the two terms. A degenerate patch, such as a flat disc, still constrains its normal. Its in-plane axes are then filled in by carrying over the consensus rotation instead of being guessed. That is the mechanism by which motion resolves the near-degeneracy problem.

Research Program

Frame predictor. A shared network outputs a frame and a reliability score per patch. It is initialized from the symmetric angular-bisector construction of LocoTrans, which we reuse because it already removes the Gram-Schmidt asymmetry. The test-time refinement idea of EquiShape optimizes local-frame vectors directly instead of moving points. We adopt that idea but apply it through the cycle loss over a sequence, not through pairwise correspondence on near-complete shapes. EquiShape's cross-attention assumes most of the shape is visible, which our inputs violate.

Patch matching. Patches are matched across frames using descriptors expressed in their own local frames, so matching is rotation-invariant by construction. Synchronizing all pairwise rotations gives every frame's pose relative to the first, up to one unknown global rotation.

Fusion and completion. The fused partial cloud goes to a rotation-invariant completion backbone. We use ESCAPE's anchor-distance encoding, which removes the remaining global rotation, though REVNET could be swapped in. Completed geometry is mapped back to each frame through the recovered poses.

Output representation. Representation heterogeneity enters as a deliberate output choice, not a headline. Unifi3D reports that signed-distance autoencoders give the highest surface fidelity. A signed distance function is a field whose value is the distance to the surface. We therefore decode both points, for comparability with the completion literature, and a signed distance function, for watertight surfaces. Inputs can be LiDAR points or back-projected depth maps without changing the pipeline.

Scope. The work targets rigid and piecewise-rigid motion first. For non-rigid motion the per-patch rotations need not agree globally. Consensus is then computed over local clusters of patches, a stretch goal tested on DeepDeform.

Benchmark. No benchmark in the file pairs arbitrary orientations, partial views and motion, so we build one. MVP and PCN shapes are driven along random rigid trajectories in front of a fixed virtual sensor. Trajectories are binned by total rotation angle, from 0° (pure translation) to 180°, and each frame is rendered as a partial scan. We also use CaSPR's rigid-motion cars sequences, and moving cars in KITTI for real-data checks.

Baselines

Do-nothing reference. Run ESCAPE on the single current frame. This is the strongest published orientation-robust completion method in the file for PCN-style data. ESCAPE reports CD-L1 = 10.58 on PCN under random ±180° rotations, against 59.31 for PoinTr. REVNET reports CD-$l_1$ = 0.89 on rotated MVP, and is the strongest single-frame equivariant competitor.

We will first reproduce those static numbers to confirm our runs are faithful. Both are then retrained on our sequence benchmark's training split with the same shapes, sensor model and GPU budget as our method. Their prediction is evaluated at every frame and also averaged across frames. Since the benchmark is new, there is no reported result found on Lacuna for any method on it. Beating them means lower Chamfer distance and higher F-score at every rotation bin above roughly 30°, with no loss at 0°.

Sequence model. CaSPR reports EMD = 11.259 on unobserved Cars frames, against 13.911 for PointFlow. It is run twice: with its ground-truth canonical supervision, as an upper reference, and with sequences presented under arbitrary orientation without that supervision. The second condition is the fair comparison to a label-free method.

Ablations. These isolate the key ingredient. We replace learned, synchronized frames with independent frames (PCA, and LocoTrans frames), keeping everything else fixed. We drop the cycle loss but keep the pairwise consensus. We substitute ground-truth poses, which shows how much of the remaining gap is registration versus completion.

Why not just…?

Why not just run REVNET or ESCAPE per frame and smooth the outputs over time? This is the most serious threat, because those methods are already strong at a single frame. Their outputs are invariant to rotation, but each is conditioned only on the side visible at that moment. Smoothing averages hallucinations of the back rather than observations of it. The rotation-bin experiment settles this directly. Per-frame-plus-smoothing should be flat across bins, while fusion should improve with coverage. If smoothing matches us at large rotation angles, the contribution reduces to a registration trick and we will say so.

Why not just register frames with ICP or PCA alignment and feed the union to a static completion network? PCA axes shift as the visible part changes; ESCAPE and REVNET both report that explicit pose normalization breaks on partial inputs. ICP needs overlap and a good initialization, and both vanish exactly when the object turns enough to reveal new sides. We will run global PCA alignment, ICP initialized from identity, and ICP initialized from our poses. We report rotation error against overlap fraction. This shows whether our frames help as an initializer, as a replacement, or not at all.

Why not just use globally equivariant latents (Vector Neuron features) and solve the relative rotation by aligning latents between frames? This is plausible and needs no local frames. However, a global latent summarizes the visible part, so two frames seeing different sides yield latents that differ beyond a rotation. Local frames on the overlapping patches do not have this problem. We implement this alternative on REVNET's encoder, using a Procrustes fit (best rotation aligning two sets of vectors) between frame latents, and compare pose error as overlap shrinks. Where latent alignment is as accurate, the local-frame machinery is unnecessary, and the experiment will tell us which regime holds.

Proposed Experiments

Coverage test. The decisive test is the curve of completion error (CD-L1, F-score of the decoded signed distance surface) against total rotation angle on the synthetic sequence benchmark, for our method versus per-frame ESCAPE and REVNET. Our expected shape is gains growing with coverage and none for translation. This follows because fusion can only add surface the sensor actually saw.

Frame stability. The second check measures what the cycle loss buys. We count axis-flip rate and angular repeatability of local frames on matched patches, split into boundary versus interior and planar versus curved patches. We compare learned-synchronized frames against PCA, LocoTrans and GFrames-style gradient frames (GFrames). The prediction is that improvement concentrates on boundary and planar patches, the documented failure cases.

Pose accuracy. Relative rotation error is reported against ground truth on the synthetic data and on CaSPR's rigid-motion cars.

Real data. KITTI moving cars have no completion ground truth; no reported number on Lacuna exists for completion there. We therefore evaluate self-consistency: completed shapes from disjoint halves of the same track should agree after pose mapping, and should agree with held-out frames' visible points.

Downstream use. A final check asks whether completed, posed moving objects help scene reconstruction. EnerGS reports KITTI PSNR 16.80 and a floater leak of 9.2% under a static-scene assumption that moving agents violate. We exclude our predicted moving-object volumes from its free-space partition per frame and measure the change in leak score and PSNR on sequences with moving vehicles. This is a test of usefulness, not a central claim.