Per-Object Static Worlds: Rigid Motion Slots Inside Feed-Forward 4D Reconstruction

Networks that rebuild a 3D scene from a short video in one pass, with no per-scene optimization, now exist. They are called feed-forward reconstructors, and they rely on one piece of old geometry: when the camera moves through a scene that stays still, the same 3D point appears in many frames, so its depth can be triangulated. Moving objects break this. The newest dynamic feed-forward models get around the problem by predicting an unconstrained 3D displacement for every pixel. They produce convincing images, but by their authors' own account the 3D shapes underneath are often wrong. We propose a different use of the most common fact about moving things: most of them are rigid. For a rigid car or chair, the frames of a video, once expressed in that object's own coordinate frame, are simply several views of a static object. If a network can guess which pixels belong to which rigid mover and how that mover is posed over time, it can triangulate each object the way it already triangulates the static background.

The core claim is that rigidity should do more than tidy up motion estimates. Inside a feed-forward model it should restore the static scene assumption one object at a time, so that multi-view evidence can be aggregated for moving objects and their geometry improves. Enforcing rigidity only on the output flow cannot provide this. It would be the first feed-forward model to use a rigid-body prior at all. It would also be the first to turn the broken multi-view consistency of moving objects, which prior feed-forward work treats only as a signal for deleting them, into a signal for reconstructing them.

What exists and where the gap is

Three feed-forward systems define the current state. BTimer predicts a frozen set of 3D Gaussians (small colored blobs rendered by splatting) at a query time in about 150 ms. It has no motion decomposition, and it states that its pixel-aligned depth yields inaccurate geometry. DGS-LRM predicts keyframe Gaussians plus dense per-Gaussian displacement vectors in about 0.6 s. Any4D predicts metric depth, camera poses and dense world-frame scene flow for 50 frames in 0.5 s. All three model motion as free per-point vectors. Our proposal keeps their speed and changes the motion parameterization and the way features are aggregated.

The rigid-prior literature supplies the method but lives outside the feed-forward regime. Weakly Supervised Learning of Rigid 3D Scene Flow collapses LiDAR flow in clusters into rigid transforms using a differentiable Kabsch step, which fits the best rotation and translation between two point sets. Doing so cut error by over 30 cm on lidarKITTI compared with fully supervised non-rigid flow. That result motivates the idea, and we reuse the Kabsch step. Multi-Body Neural Scene Flow adds a distance-preservation regularizer to per-scene optimization. It lowers Argoverse error from 0.090 m to 0.033 m but doubles runtime to about 186 s per frame pair. It also reports a constraint on our design: rigidity losses can pull moving objects toward zero motion, which is trivially rigid. R5DGS propagates one rigid transform per object across 4D Gaussians. It needs per-scene optimization and SAM2+DEVA 2D trackers, and it reports a photometric cost on non-rigid regions and moving shadows. That cost is the second constraint on our design. Split4D shows that grouping by 3D trajectory rather than by 2D trackers raises instance mIoU from 0.592 to 0.893. We take this as evidence that motion itself can drive segmentation, though Split4D is optimization-based and uses linear velocities rather than full rigid motion.

On the static-scene side, pixelSplat resolves depth by attending along epipolar lines. Epipolar lines are the lines on which a point's match must lie if the scene is static. DGGS uses the failure of this consistency to find and prune moving distractors. We invert DGGS's use of the signal. Pixels that are inconsistent under the camera's motion but consistent under some other rigid motion form an object to be reconstructed in its own frame.

Core mechanism

The network predicts, for each pixel $i$ of the context frames, a depth, Gaussian appearance parameters, and a soft assignment $w_{ik}$ over $K$ motion slots. Slot 0 is reserved for the static world and moves only with the camera. Each other slot $k$ carries a rigid trajectory $(R_k(t), T_k(t))$, a rotation and translation for every frame time $t$. A Gaussian with world position $x_i$ at the reference time moves as

$$x_i(t) = \sum_{k=0}^{K} w_{ik}\,\big(R_k(t)\,x_i + T_k(t)\big) + r_i(t),$$

where $r_i(t)$ is a small per-point residual for non-rigid deformation, articulation and shadows. The residual is penalized in proportion to its magnitude so the model uses it only where rigid slots fail. This addresses the photometric degradation R5DGS reports without giving up rigidity where it holds.

What is new is what the slots do inside the network. The network runs a few refinement rounds. After each round, every frame's features are warped into each slot's canonical frame using the current slot poses, the assignments and the current depths. Cross-frame attention then runs within each slot along that slot's own epipolar geometry, treating the object as static. For slot 0 this is ordinary static multi-view reasoning in the style of pixelSplat. For a moving car it is multi-view reasoning over the car's appearances across time. The refined features update depth, assignments and slot poses. Poses are re-estimated in closed form by a weighted Kabsch fit on the predicted correspondences, which keeps them rigid by construction and differentiable end to end.

Training uses synthetic data with ground-truth motion and instances, namely Kubric (as in DGS-LRM), PointOdyssey and Dynamic Replica, mixed with the static corpora BTimer pretrains on, where every pixel should land in slot 0. Supervising motion directly with ground-truth trajectories and rendering losses removes the zero-motion collapse that Multi-Body Neural Scene Flow suffers under a purely self-supervised Chamfer objective. Slot assignments are supervised only through motion. Instance labels are held out and used purely for evaluation, so segmentation must emerge from motion as it does in Split4D.

Research Program

The work has three parts. The first builds the model on a shared video-transformer backbone and trains it under the same data and compute as a matched unconstrained-flow variant. The second tests the central causal claim that per-slot canonical aggregation improves the geometry of moving objects, as distinct from merely regularizing their motion. We measure depth error restricted to dynamic pixels on Dynamic Replica and held-out Kubric scenes, where ground-truth depth exists. We report it separately for rigid and non-rigid instances. The third characterizes where the prior helps and hurts. That analysis covers rigid-dominated scenes versus human- and animal-dominated ones, the number of slots $K$, the magnitude of the residual, and objects that enter mid-sequence, the failure mode flagged for Any4D's first-frame anchoring.

Baselines

The naive reference points are two. The first is our backbone with all motion set to zero, meaning a static feed-forward reconstructor pretrained as BTimer is. The second is a zero-flow predictor on the motion metrics. Any method must clear both by a wide margin on dynamic regions, or the benchmarks are too static to be informative. The Reality Check paper shows this happens often.

The strongest existing competitor for motion is DGS-LRM. It is the only feed-forward Gaussian model with dense trajectories evaluated on PointOdyssey, where it reports an ATE-3D of 0.21 against SpatialTracker's 0.22. On DyCheck it reports a PSNR of approximately 14.89. For dense metric 4D geometry the strongest competitor is Any4D, with an Average Point Distance (APD) of 93.44 on Dynamic Replica against 62.34 for the next baseline. For rendering, BTimer reports 25.82 dB on the NVIDIA Dynamic Scene dataset. We will compare against these published numbers under their own protocols. More importantly, we will retrain an unconstrained per-pixel-flow head, DGS-LRM-style, on our exact backbone, data mixture and training budget. That matched model is the fair competitor. Beating it means lower ATE-3D on PointOdyssey and lower dynamic-region depth error. It also means lower rigidity violation, measured as drift in pairwise distances within ground-truth rigid instances. Masked PSNR must not drop.

Three ablations isolate the key ingredient. The first applies post-hoc rigidification: clustering plus Kabsch fitting on the unconstrained model's output flow at test time. The second trains the unconstrained model with an as-rigid-as-possible distance loss. The third keeps our slot parameterization but removes canonical re-aggregation, so that slots only shape the output. The claim stands only if the full model beats all three on dynamic-region geometry. If only the output-parameterization ablation matches it, the contribution collapses to a motion regularizer.

Why not just…?

Why not just run clustering and Kabsch fitting on DGS-LRM's flow after the fact? This is the most serious threat. On motion metrics for rigid objects it may nearly match us, because it inherits the rigidity benefit shown in Weakly Supervised Learning of Rigid 3D Scene Flow. Post-hoc fitting cannot change the depths the network already committed to, though, and BTimer's stated weakness is in those depths. The post-hoc ablation directly tests whether geometry improves only when rigidity enters the forward pass. If dynamic-region depth error is statistically indistinguishable between the two, we will report that the post-hoc pipeline is sufficient.

Why not just add an as-rigid-as-possible loss to an unconstrained flow head? Such losses are cheap and have worked in optimization settings. Multi-Body Neural Scene Flow shows they fight the data term and can drive motion toward zero. In a feed-forward model they shape outputs without giving the network a way to pool evidence across time for a moving object. The loss ablation, trained under an identical budget, measures this.

Why not just segment with SAM2 and DEVA, then fit rigid motion per mask? R5DGS uses this route and reports that it breaks under occlusion, fast motion and crossing objects. Split4D's jump from 0.592 to 0.893 mIoU when grouping by 3D trajectories instead suggests motion-based grouping is stronger. We will run the two-stage pipeline on our dynamic benchmarks and report instance mIoU and trajectory error, stratified by occlusion level.

Why not just predict frozen snapshots per timestamp, as BTimer does? Snapshots render well, and the Reality Check paper found that simple time-conditioned models often beat elaborate deformation models on image quality. Snapshots provide no trajectories, no decomposition and no extrapolation, however, and their geometry is admittedly inaccurate. We compare BTimer on the NVIDIA dataset for rendering and on dynamic-region depth for geometry. If our model wins only on the latter, that is the honest outcome and still the one this proposal targets.

Proposed Experiments

Training uses the Kubric, PointOdyssey and Dynamic Replica mixture, with RealEstate10K and the other static corpora serving as slot-0 data. Evaluation covers PointOdyssey for ATE-3D, Dynamic Replica for APD and dynamic-region depth, held-out Kubric for emergent instance mIoU against ground-truth instances, the NVIDIA Dynamic Scene dataset for PSNR, and DyCheck iPhone under the Reality Check co-visibility-masked protocol for masked PSNR and PCK-T. Neither BTimer nor DGS-LRM has been reported under that strict protocol, so we will rerun both where code allows.

The decisive check is the comparison of dynamic-region depth error between the full model and the model without canonical re-aggregation. It is reported separately for ground-truth rigid instances, where we expect a clear gain, and non-rigid ones, where we expect parity thanks to the residual term. A gain on rigid objects that is absent in the post-hoc and loss ablations would confirm that rigidity serves as a per-object static-world assumption rather than a cosmetic constraint. A loss on DyCheck's largely non-rigid scenes beyond one standard deviation of the matched baseline would mean the residual term is too weak. We would respond by revisiting its penalty rather than narrowing the claim. Runtime will be reported against DGS-LRM's 0.6 s and BTimer's 150 ms. The added cost of a few Kabsch solves and slot-wise attention should stay within the same order of magnitude, far from the roughly 186 s per pair of optimization-based rigid methods.