Rigid Movers as a Calibration Signal for Uncalibrated 4D Reconstruction

When a few cameras are set up casually around a scene, such as phones on tripods, a robot cell, or a street corner, nobody knows their focal lengths or relative poses. Anything that moves in front of them is usually treated as a nuisance for recovering that geometry. This proposal argues the opposite. Every object that moves rigidly, like a box being carried, a car, a robot link, or a human forearm, supplies a free calibration measurement. The measurement becomes readable if the scene is first reconstructed in a coordinate frame defined only by image matches, a "projective frame": one that preserves straight lines but not distances or angles. In that frame a rigid 3D motion appears as a 4×4 matrix. That matrix is the true rotation and translation seen through one unknown distortion, and the distortion is shared by every moving object at every moment. Pooling many such motions recovers the distortion, which amounts to metric calibration. The same fit also separates the static background from each moving part and gives a dense, time-varying 3D model.

The classical pieces already exist but have never been joined. Devernay and Faugeras upgraded a projective reconstruction to metric using the rigid displacements of a moving stereo rig viewing a static scene. Demirdjian and Horaud used the same 4×4 projective motion matrices to separate moving objects from a static background, but only for sparse points over two stereo time steps, and they discarded the movers. Modern dynamic reconstruction goes one of two ways. Some methods require calibration, including Split4D, RobustSplat and DyNeRF. Others avoid calibration either by abandoning multi-view geometry for monocular depth priors, as in StreamSplat, or by requiring a specific object class with a body model, as in Humans as a Calibration Pattern. We propose using arbitrary rigid or piecewise-rigid motion as the calibration target inside a dense 4D Gaussian splatting model.

Core Claim

Consider synchronized or roughly synchronized video from two to four static, uncalibrated cameras. The static background fixes the epipolar geometry between the cameras, meaning the relation that tells where a point in one image can appear in another. That geometry defines a projective reconstruction frame without estimating any metric camera matrix. This is the "implicit basis" of the title: a coordinate frame fixed entirely by correspondences. In this frame, the motion of rigid part $k$ from time $t$ to $t'$ is a $4\times4$ collineation, a straight-line-preserving map, written $\mathbf{H}_{k,t\to t'}$. It satisfies

$$ \mathbf{H}_{k,t\to t'} \simeq \mathbf{U}^{-1}\,\mathbf{D}_{k,t\to t'}\,\mathbf{U}, $$

where $\mathbf{D}_{k,t\to t'}$ is the unknown Euclidean rotation and translation, $\mathbf{U}$ is the single projective-to-metric upgrade shared by all parts and times, and $\simeq$ means equality up to scale. This is the conjugacy of Devernay and Faugeras, with the roles of rig and scene exchanged. The upgrade $\mathbf{U}$ has 8 unknowns: 3 for the plane at infinity, the plane that parallel lines meet at in the projective frame, and 5 for the intrinsics. Each rigid motion with a new rotation axis adds constraints. With two static uncalibrated views, the background alone cannot reliably determine metric structure: focal lengths from a single fundamental matrix are poorly conditioned. We therefore expect scene dynamics to supply exactly the information static structure-from-motion lacks in few-camera setups.

The contribution is to make this relation the organizing constraint of a dense dynamic model. We parameterize every moving cluster's trajectory as $\mathbf{U}^{-1}\mathbf{D}\,\mathbf{U}$ with $\mathbf{D}$ restricted to rigid motions. The photometric rendering loss from every moving cluster then sends gradient into the one shared calibration. Calibration, motion segmentation and 4D reconstruction become a single fit.

Research Program

Stage one: sparse projective structure. We estimate fundamental matrices between the static cameras from matches that stay stationary over time. Following Rothwell et al., we triangulate tracked points in the canonical projective frame with over-constrained linear least squares, solved by singular value decomposition. We avoid image-plane invariants because that comparison found cross-ratio methods an order of magnitude less stable (error 154.5 versus 11.1 at 0.5-pixel noise). This is the constraint the classical literature places on our design: the frame is implicit, but the arithmetic inside it must be over-determined.

Unknown time offsets. If the cameras are not synchronized, a moving point violates the background's epipolar geometry in proportion to the time offset. We therefore choose per-camera offsets by minimizing the epipolar residual of dynamic tracks. This replaces the human-pose alignment step of Humans as a Calibration Pattern with a check that works for any mover.

Stage two: motion discovery and upgrade. Points that fail the identity collineation, which is the static background, are grouped by sequential robust fitting of 5-point projective motion matrices. This generalizes Demirdjian and Horaud from two time steps to long tracks and many clusters. We then solve for $\mathbf{U}$ jointly over all cluster-time pairs. Standard mild priors on the intrinsics (zero skew, principal point near the image centre) are used only as regularizers. Known degeneracies are handled explicitly. Pure translations carry no intrinsic information, and all rotations about parallel axes, as with cars turning on a flat road, leave a one-parameter family of solutions. We detect these cases from the spread of rotation axes and fall back on the priors, reporting the remaining ambiguity rather than hiding it.

Stage three: dense 4D Gaussians. 3D Gaussian splatting represents a scene as many small coloured ellipsoids rendered by projection. We initialize static and per-cluster Gaussians from the triangulated tracks. Each cluster carries an SE(3) trajectory, meaning a rigid rotation-plus-translation path, plus a small non-rigid residual to absorb non-rigid parts such as clothing. Cameras are rendered through $\mathbf{P}_i\mathbf{U}^{-1}$, where $\mathbf{P}_i$ is camera $i$'s projective camera matrix, and the upgrade $\mathbf{U}$ is refined photometrically together with everything else.

RobustSplat identified runaway splitting and cloning of Gaussians, driven by transient photometric errors, as the failure mode on dynamic content. We address it differently. A pixel whose error is well explained by an existing cluster's rigid motion is assigned to that cluster instead of triggering new Gaussians. New Gaussians are created only for residuals that no rigid motion explains.

Baselines

The naive reference is static-only calibration. It upgrades the background projective reconstruction using the fundamental matrices plus the same intrinsic priors, with no use of dynamics, and then trains the identical Gaussian model with dynamic content discarded by a robust loss. This isolates how much the movers contribute. The key ablation feeds motion clusters into segmentation and densification but freezes $\mathbf{U}$ at the static-only estimate. That separates "motion helps segmentation" from "motion calibrates."

The strongest existing competitor is Humans as a Calibration Pattern. It is the only method in the file that recovers calibration and synchronization from dynamic content across uncalibrated videos, using SLAHMR body fits, dynamic time warping and Procrustes alignment, followed by K-Planes refinement. On CMU Panoptic it reports initialization errors below 5.5° rotation, below 26 cm translation and 1.34 frames of time offset. After refinement it reports rotation as low as 0.4°, translation of 0.22 cm and time offset below 0.03 frames. We will run the released or re-implemented pipeline on the same camera subsets and the same desynchronization protocol, with matched refinement compute.

On Panoptic, beating it means matching its refined numbers without any body model. On scenes without humans, where it cannot run, the question is how close we come to ground-truth calibration. Our stage-two initialization should beat its initialization errors, because rigid-part collineations are exact where body-model fits are approximate.

The second competitor is StreamSplat. It is an uncalibrated dynamic Gaussian model built on DepthAnythingv2 and DINOv2, reporting 37.83 keyframe PSNR on DAVIS. We run it per camera and compare novel-view quality at held-out cameras, because it never recovers inter-camera geometry. The third competitor is the calibrated pipeline: COLMAP on masked background with the same Gaussian model. This is run with ground-truth intrinsics as an upper bound and with default intrinsics as a realistic comparison.

Why not just…?

Why not just run COLMAP on the background with segmentation masks? With dozens of cameras this works. The Plenoptic rig is itself calibrated this way, so we treat that condition as an upper bound, not a competitor we expect to beat. The claim is limited to two to four cameras with wide baselines, where static structure-from-motion with unknown focal lengths is poorly constrained. Mask drift also corrupts the geometry, as Split4D documents for 2D trackers. An experiment that varies the camera count from two up to all available will show where the crossover lies. If COLMAP matches us already at three cameras, the method's useful range is narrow, and we will say so.

Why not just fix scale and focal length with monocular metric depth? Networks like ScaleDepth, or the DepthAnythingv2 prior inside StreamSplat, can supply metric depth without any geometry. This is a real threat on typical indoor scenes. StreamSplat itself reports that such priors fail on thin structures and strong perspective, but that failure may be rare in practice. We include "upgrade from monocular depth" as a baseline: fit $\mathbf{U}$ by aligning projective depths to predicted depths. We also report results separately on the thin-structure and close-range subsets where the prior is expected to break.

Why not just treat movers as outliers? RobustNeRF and RobustSplat do this and get a clean background: 30.87 dB versus 25.22 dB for Mip-NeRF 360 on the RobustNeRF distractor data. However, they assume calibrated cameras and output no dynamic model. The static-only naive baseline is this alternative moved into the uncalibrated setting. If it recovers calibration as well as ours, the core claim fails.

Proposed Experiments

Calibration accuracy with controlled motion. A synthetic rendered benchmark provides exact ground truth. We vary the number of rigid movers, the diversity of their rotation axes (including the degenerate planar case), image noise, and the number of cameras. We plot rotation, translation and focal-length error against the number of independent motions. Because each non-degenerate motion adds constraints on the 8-parameter upgrade, error should fall roughly as the inverse square root of the number of motions, the usual averaging of independent estimates. This holds until noise in estimating the collineations dominates.

Real footage. On CMU Panoptic we compare directly against Humans as a Calibration Pattern using its reported metrics. On the Plenoptic Video dataset of DyNeRF, we use subsets of two to four GoPro cameras, its provided calibration as ground truth, and the held-out cameras for novel views. We report PSNR, SSIM and LPIPS; the file lists no reported PSNR for this dataset, so our calibrated-COLMAP run provides the reference. On the multi-human data of Split4D we evaluate static-versus-dynamic decomposition by mIoU against Split4D's reported 0.893, noting that Split4D is calibrated and uses 2D masks.

Decisive checks. Three outcomes would falsify the proposal. The first is if freezing $\mathbf{U}$ at the static-only estimate gives calibration and novel-view quality statistically indistinguishable from the full method on the two- and three-camera subsets. The second is if the monocular-depth upgrade matches ours on the thin-structure and close-range subsets. The third is if our human-free pipeline lands no closer to Panoptic's ground truth than the body-model initialization, whose errors are 5.5° rotation and 26 cm translation.