Reconstruction methods that recover 3D scenes from casual video without known camera positions need two things at once. They must throw away pixels that belong to moving things such as people, cars or shadows, and they must keep enough stationary background to work out where the camera was. Current systems handle these separately. A segmentation or outlier step first decides what is moving. Pose estimation then uses whatever is left. When a pedestrian fills the middle of the frame, what is left can be a thin strip of building edge in one corner. Structure-from-motion has long shown that a small patch of support like that makes camera rotation and translation nearly impossible to tell apart (Structure-from-Motion Revisited). A masked solver still returns a pose in that situation, but the pose is wrong and nothing warns that it is wrong.
We propose deciding what to ignore by asking two questions together: how likely is this image patch to be moving, and how much does it help pin down the camera? Each frame's static/moving weights are chosen by a small convex problem. The problem maximizes how well-determined the 6-degree-of-freedom camera pose is, measured by the log-determinant of a 6×6 information matrix, and it pays a penalty for trusting patches that look transient. A patch that looks somewhat suspicious but is the only thing constraining one rotation axis is kept at reduced weight rather than deleted. When even the best selection leaves the pose poorly determined, the update is damped toward a motion prior instead of being trusted. In plain terms, this borrows "optimal experimental design" from statistics and applies it to distractor masking inside pose-free reconstruction.
Core Claim
The failure of pose-free reconstruction on dynamic scenes has two parts. One is not knowing which pixels move. The other is that removing the moving pixels can leave the pose unobservable. The field's uniformity tools do not address the second part. Grid bucketing, farthest-point sampling and density reweighting (DKM, CS-RANSAC) all redistribute weight among points already declared static, so they cannot restore leverage that the mask removed. We claim three things. First, pose error in masked pose-free pipelines is predicted by the smallest eigenvalue of the pose information matrix built from the retained support, and it is poorly predicted by how much of the frame was masked. Second, choosing mask weights to maximize that information, subject to a transient penalty, reduces trajectory error specifically in the degenerate-support regime. Third, the method matches ordinary masking when the background is well spread, which is how it should behave.
Where this sits relative to prior work
Two groups of prior work motivate the proposal, and each covers only half of the problem.
The first group removes distractors but assumes camera poses are already known from COLMAP:
- RobustNeRF trims the worst half of residuals within 16×16 neighborhoods.
- NeRF On-the-go predicts per-pixel uncertainty from DINOv2 features.
- RobustSplat delays Gaussian densification and bootstraps masks from coarse to fine.
- 3DGS-HPC classifies 16×16 patches by calibrating perceptual error against a mixture model fitted to photometric error.
With poses fixed, over-masking a blank wall costs these methods little. The research file's Gap 3 notes that the same over-masking in a pose-free system removes exactly the anchors the trajectory depends on. We reuse 3DGS-HPC's patch-level transient score as our "how suspicious" input. The difference is that our masking decision is no longer made independently of pose.
The second group optimizes pose but assumes a static scene. NoPe-NeRF jointly fits a neural radiance field, poses and the scale and shift of monocular depth maps. Look Gauss, No Pose puts analytic pose gradients into the 3D Gaussian Splatting (3DGS) rasterizer and adds an anisotropy penalty against needle-shaped Gaussians. 3DGS represents a scene as many small colored 3D blobs rendered by projection. We build on the Look Gauss machinery, because its per-pixel pose gradients are exactly what we need to assemble information matrices cheaply.
The closest existing systems are 4D3R and RoDyGS, which do both halves.
- 4D3R segments moving objects with SAM2 and runs masked PnP-RANSAC on MonST3R geometry. PnP-RANSAC solves camera pose from 2D–3D matches while rejecting outliers.
- RoDyGS refines poses against a static Gaussian set that is separated with user help.
Both treat the mask as fixed input to pose solving. Neither checks whether the surviving static support can constrain the pose. That missing check is the gap this proposal targets.
Free-DyGS and PAD3R constrain the scope. The first relies on stereo depth in surgical video. The second is object-centric. We therefore target monocular, scene-centric capture.
Method
We work inside a pose-free 3DGS pipeline, initialized like 4D3R from pairwise point-map predictions.
Information matrix. For frame $t$, split the image into 16×16 patches indexed by $p$. For each patch, compute:
- $s_p \in [0,1]$, a transient score from the patch classifier;
- $H_p = \sum_{x \in p} g_x g_x^\top$, the 6×6 accumulated outer product of the per-pixel photometric pose gradients $g_x$.
$H_p$ approximates how much patch $p$ tells us about each direction of camera motion. The rasterizer already produces the gradients $g_x$, and summing their outer products per patch adds a small kernel.
Weight selection. Patch weights $w_p \in [0,1]$ are chosen by
$$ w^\star = \arg\max_{w \in [0,1]^P} \; \log\det\Big(\sum_p w_p H_p + \epsilon I_6\Big) \;-\; \lambda \sum_p w_p\, s_p , $$
where $P$ is the number of patches, $\epsilon$ is a small ridge term, $I_6$ is the 6×6 identity and $\lambda$ sets how costly it is to trust suspicious patches. The log-determinant is concave in $w$ and the penalty is linear, so this is a convex problem over a few hundred variables with a 6×6 matrix. A few projected-gradient steps per frame solve it.
How this differs from uniformity heuristics.
- Spread in the image is rewarded only when it adds information along pose directions that are otherwise weak. A second patch next to an existing one adds little.
- Sky and blank walls contribute near-zero $H_p$, so they get no credit. This avoids the known flaw of uniform random pixel sampling.
- A shadowed but textured facade can be retained at partial weight when it is the only constraint on some direction.
Gated pose update. The weights $w^\star$ scale the photometric loss for both the pose update and the Gaussian update. If the smallest eigenvalue of $\sum_p w^\star_p H_p$ falls below a threshold $\tau$, the pose step is blended toward a constant-velocity prediction in proportion to the deficit. The reconstruction therefore does not absorb an unobservable pose error into distorted geometry, which is the ambiguity failure that Look Gauss's anisotropy loss fights only indirectly.
Moving content. The moving part of the scene is modeled with a standard deformation field on a separate set of Gaussians. That component is not our contribution, and it is held fixed across all compared methods.
Research Program
Analysis phase. First we test the premise before building anything, on ScanNet (ScanNet).
- ScanNet sequences have sensor-derived ground-truth poses.
- We composite moving distractors (segmented people and objects pasted with consistent motion) at controlled coverage fractions and controlled positions: center, one side, or everything except one corner.
- We run a masked pose-free baseline with oracle masks and record Absolute Trajectory Error (ATE) against the smallest eigenvalue of the retained information matrix.
If ATE does not track that eigenvalue better than it tracks masked area, the premise is wrong and we stop.
Building phase. If the premise holds, we implement the selection and gating inside the Look Gauss-style rasterizer. We then study three things: sensitivity to $\lambda$ and $\tau$, how often the gate fires, and whether soft retention of suspicious patches leaks ghosting into the static reconstruction.
Baselines
Naive reference. Look Gauss, No Pose run unchanged on dynamic video, with no masking. It shows how badly static-assuming pose gradients are pulled by distractors.
Robust-loss reference. The same pipeline with RobustNeRF-style patch-trimmed loss. Its reported 30.87 dB PSNR versus 25.22 dB for Mip-NeRF 360 on distractor scenes was obtained with known poses. That establishes what trimming achieves when pose is not the bottleneck.
Strongest existing method: 4D3R. It combines SAM2 masks with motion-aware bundle adjustment. The file reports "up to a 1.8dB PSNR improvement" on HyperNeRF and roughly 50 minutes of training. It reports no ATE, so no reported result was found on Lacuna for 4D3R trajectory error.
RoDyGS. The file lists no numbers for it on Kubric-MRig or the iPhone dataset, so no reported result was found on Lacuna for those either. We will run it ourselves with automatic rather than user-provided separation, and state that change explicitly.
Fairness. All methods share:
- the same MonST3R initialization;
- the same dynamic-Gaussian component;
- the same number of iterations and GPU;
- equal hyperparameter search budgets (a fixed grid of equal size per method) tuned on held-out sequences.
Ablations that isolate the key ingredient. Each keeps our transient scores and pipeline and swaps only the weighting rule:
- hard thresholding of $s_p$ (standard masking);
- hard masking followed by grid top-$k$ bucketing;
- hard masking followed by DKM-style inverse-density reweighting;
- our information-driven weights without the gate;
- the gate without the weight optimization.
What counts as winning. We must lower ATE relative to the best ablation in the degenerate-support bins of the controlled sweep, with confidence intervals across sequences that do not overlap. We must also stay within noise of it in well-spread bins. On standard benchmarks, we must match or beat 4D3R's rendering quality at comparable training time.
Why not just…?
Why not just bucket the surviving static features into a grid, as ORB-style trackers and CS-RANSAC do? This is the most serious threat. When the static background still covers the frame, bucketing probably works as well as our method, and we expect the controlled sweep to show that. Bucketing can only reorder points that survived the mask. It cannot bring back a patch the mask removed. It also treats a well-textured corner and an uninformative sky cell as equally valuable. CS-RANSAC's gains of 3–85% pixel error were measured on static planar homographies where support was never removed. The corner-only occlusion bins of the sweep decide the question. If bucketing plus hard masking matches us there, our contribution reduces to the gate alone, and we will report that.
Why not just segment people and cars with SAM2 and then run NoPe-NeRF or Look Gauss? That pipeline is essentially 4D3R, and it is our main baseline. The file also records that semantic segmenters miss shadows and shifted furniture (3DGS-HPC), and that video masks drift under occlusion (Split4D). Even with perfect masks, the analysis phase uses oracle masks precisely to show that masked-out support can leave pose unobservable. Our method uses such masks as one input and does not replace them.
Why not just use robust losses such as trimmed least squares during joint optimization? Trimming decides by residual size alone. The file notes that it discards sharp static texture, whose early residuals are large. Trimming also has no notion of which pose directions it is starving. The robust-loss baseline measures this directly, and we expect it to fail in the same degenerate bins as hard masking.
Why not just run COLMAP with outlier rejection first? The file records COLMAP failing when moving objects dominate the frame (4D3R) and on casual texture-poor video (NoPe-NeRF). We will still include COLMAP poses followed by RobustSplat as a reference on NeRF On-the-go. There, COLMAP poses are also the evaluation reference, so we report its registration failures rather than its ATE.
Proposed Experiments
Controlled sweep. The ScanNet sweep is the decisive test because it has true poses and lets us control where support disappears. The metrics are:
- ATE and relative rotation error, binned by the smallest eigenvalue of the retained information;
- PSNR, SSIM and LPIPS on held-out static views.
Real distractor data. We use NeRF On-the-go and the RobustNeRF benchmark in pose-free mode.
- We evaluate static-scene PSNR after test-time pose alignment.
- Anchors from known-pose methods are NeRF On-the-go's 21.41 dB on Patio and RobustNeRF's 30.87 dB.
- We ask how much of the gap between pose-free and posed methods our weighting closes.
Fully dynamic scenes. We use Kubric-MRig (synthetic ground-truth poses, for ATE), the iPhone and NVIDIA dynamic datasets, and HyperNeRF (for dynamic novel-view PSNR and LPIPS against 4D3R and RoDyGS).
Two decisive checks.
- On real footage, frames where the gate fires should be the frames where the baselines' trajectories visibly jump.
- Replacing $\log\det$ with the trace of the same matrix should lose most of the benefit. The trace rewards a pile of redundant patches, whereas the log-determinant rewards covering weak directions. This check shows that the gain comes from observability and not from simply keeping more pixels.
References
- Structure-from-Motion Revisited
- DKM: Dense Kernelized Feature Matching for Geometry Estimation
- RANSAC versus CS-RANSAC
- RobustNeRF: Ignoring Distractors with Robust Losses
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior
- Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos
- RoDyGS: Robust Dynamic Gaussian Splatting for Casual Videos
- Free-DyGS: Camera-Pose-Free Scene Reconstruction based on Gaussian Splatting for Dynamic Surgical Videos
- PAD3R: Pose-Aware Dynamic 3D Reconstruction from Casual Videos
- ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation