Methods that rebuild a static 3D scene from photos taken while people and objects move through it all rely on one trick. After fitting a model, anything the model cannot reproduce consistently across views is treated as a passing "distractor" and ignored. That trick quietly assumes the camera positions are exact. When the cameras are slightly wrong, which happens with a mount that vibrates, a rig that drifted, or poses estimated from blurry video, the permanent background also stops matching across views. The distractor filter then deletes real walls, edges, and thin objects along with the pedestrians. We propose to separate the two causes before filtering. A small camera error leaves a specific signature in the reconstruction error: it is spread across the whole image, it is concentrated on image edges, and it can be undone by a six-parameter shift of the camera. A moving object leaves an error confined to one region that no camera shift can explain. Our method explains away as much of the error as a per-view camera correction can, and only then decides what is a distractor.
The opportunity is practical. The calibration literature shows how brittle geometry-based vision is to tiny angular errors. In Calibration-free BEV Representation for Infrastructure Perception, a projection-based roadside detector falls from 29.2% to 16.9% mAP at 0.5° of rotation noise and to 0.3% at 5°. The distractor-removal literature, meanwhile, evaluates only with clean poses. Nobody has measured what happens to distractor-free reconstruction when the camera is slightly wrong, or built a filter that stays correct in that regime.
Core Claim
Residual-based distractor filters confuse camera error with scene motion, and the confusion cannot be repaired simply by making the poses learnable. The reason is circular. Robust losses throw away the highest-error pixels, and under camera error those pixels are exactly the image edges that carry information about the pose. The pose optimizer is therefore starved of the evidence it needs to correct itself. We claim that a pose-attribution step fixes this and recovers most of the clean-pose reconstruction quality under realistic jitter (0.1° to 2°). The step is a per-view robust fit of a small camera correction, computed on all pixels with a bounded rather than trimmed loss, and run before any pixel is labelled a distractor. It should also preserve rarely-seen static structure that current filters erase.
Relation to Prior Work
Three distractor-removal methods motivate the problem and supply the base system, and all three assume exact poses.
- RobustNeRF trims the 50% highest-error residuals and adds a smoothness prior over 16×16 patches.
- RobustSplat delays the growth of new Gaussians (the small 3D blobs that Gaussian Splatting renders) so that transient objects do not get baked into the scene. It also bootstraps distractor masks coarse-to-fine.
- 3DGS-HPC classifies 16×16 patches using photometric error plus DINOv2 features. Its authors document a "rare object problem": static surfaces seen in only a few views are misread as distractors.
We keep their filtering logic and change what the filter is allowed to see. Our argument is that camera error turns ordinary static surfaces into "rare objects" wholesale.
A second group of methods jointly estimates camera motion and scene dynamics. These show that pose and scene can be refined together, which constrains our design.
- UPGS learns camera poses inside Gaussian Splatting with a scene-only, then pose-only, then joint schedule. It needs a rough initial trajectory.
- Free-DyGS solves pose and tissue deformation frame by frame, but requires stereo depth.
- VDNeRF lets a static radiance field alone drive pose estimation, while a dynamic field absorbs moving content.
- Back on Track converts moving points into "pseudo-static" landmarks for bundle adjustment. It reports 0.034 absolute translation error on MPI Sintel.
None of these pairs pose refinement with a trimmed or patch-classifying distractor filter, and none studies how the two interact. We adopt UPGS's staged schedule as a baseline ingredient, not as our contribution.
No Parameters, No Problem (TrackGS) optimizes intrinsics and extrinsics from feature tracks. Its authors state that moving objects corrupt those tracks, which is exactly the failure we target from the other side.
Research Program
The system starts from a standard Gaussian Splatting optimizer with a RobustSplat-style distractor pipeline. Each training view $i$ carries a learnable pose correction $\delta\xi_i \in \mathbb{R}^6$ on top of its noisy input pose.
At every mask update, before computing distractor labels, we linearize the rendered image around the current pose and solve
$$\delta\xi_i^{*} = \arg\min_{\delta\xi} \sum_{p} \rho\big(r_i(p) - \nabla I_i(p)^{\top} J_i(p)\,\delta\xi\big),$$
where the symbols are as follows:
- $r_i(p)$ is the photometric residual at pixel $p$.
- $\nabla I_i(p)$ is the rendered image gradient.
- $J_i(p)$ is the 2×6 Jacobian of the pixel's projected location with respect to the camera pose. It uses the rendered depth, so translation effects correctly shrink with distance.
- $\rho$ is a Huber penalty. It bounds the influence of large residuals rather than discarding them.
The correction is applied, the image is re-rendered, and only the remaining residual feeds the trimming or patch-classification rule. Moving objects produce large residuals uncorrelated with $\nabla I^{\top} J$. They pull on $\delta\xi$ with bounded force and survive as residual. Camera error, by contrast, is absorbed.
We add a guard that directly targets the rare-object problem. A patch whose residual dropped by more than a set fraction under the correction is exempt from distractor labelling for that round.
The program has three parts. The first is a controlled measurement study. We inject per-view rotation and translation noise, following the rotation-noise protocol of CBR, into standard distractor benchmarks and chart where RobustNeRF-, RobustSplat-, and 3DGS-HPC-style filters begin erasing static structure. The second is the attribution method itself. The third is an analysis of when attribution fails. We expect failures when a large rigid distractor, such as a bus filling half the frame, is itself well explained by a camera shift. We will quantify that confusion as a function of how much of the image the distractor covers.
Baselines
The naive reference is vanilla 3D Gaussian Splatting trained on the noisy poses with no distractor handling. It bounds how bad things get.
The strongest existing distractor methods are RobustSplat and 3DGS-HPC, each run on the same noisy poses with their published settings. They are the right competitors because they are the best-performing filters in the file. 3DGS-HPC reports a 1.39 to 5.54 dB PSNR gain over native 3DGS on the RobustNeRF benchmark, and RobustNeRF reports 30.87 dB against 25.22 dB for Mip-NeRF 360 on its Robotics Lab scenes. All of these were measured with clean poses. We found no reported result on Lacuna for any distractor-free method under pose noise, so the first measurements under noise will be ours.
The most dangerous competitor is RobustSplat with learnable per-view poses trained on UPGS's three-stage schedule. It tests whether ordinary joint optimization already solves the problem. Every method will get the same noisy poses, the same iteration count, the same Gaussian budget, and the same hyperparameter search budget, tuned on one held-out scene per benchmark.
Two further reference points bracket the results. An oracle RobustSplat trained on clean poses is the upper bound. TrackGS is a pose-free static method that shows what happens when poses are re-estimated while ignoring dynamics.
Ablations isolate the key ingredient in three ways:
- Attribution run with a trimmed loss instead of Huber, which should reproduce the starvation effect.
- Attribution run after mask computation instead of before.
- Attribution with the rare-object guard removed.
Beating the learnable-pose baseline means two things. First, a statistically reliable PSNR gain on held-out clean views at 0.5° and above. Second, higher precision on the static-versus-transient labels, with no loss at zero noise.
Why not just…?
Why not just make poses learnable inside RobustSplat? This is the real threat, and the proposal is built so that it can lose to it. Our hypothesis is that trimming removes the edge pixels that carry the pose gradient, so joint optimization stalls at a biased pose while static edges are masked away. The trimmed-versus-Huber ablation and the learnable-pose baseline test this directly. If learnable poses match our method at every noise level, we will report the measurement study as the contribution and drop the attribution claim.
Why not just mask moving objects with a 2D segmenter and then refine poses? Segmenters miss objects outside their categories. 3DGS-HPC documents that they miss shadows and moved chairs. They also over-mask stationary instances of moving classes: VDNeRF shows parked cars wrongly removed, costing pose landmarks. We will include a segmentation-mask-plus-pose-refinement variant on NeRF On-the-go to measure this rather than assume it.
Why not just recalibrate more carefully, or rerun structure-from-motion per capture? Structure-from-motion itself breaks on blur and moving content, as documented by UPGS and TrackGS. Vibration-induced jitter also changes from frame to frame, so a one-off calibration cannot remove it. Our noise sweep includes a very small noise level to show where recalibration would already suffice.
Why not just loosen the distractor threshold? The file notes the tradeoff in RobustSplat: relaxed thresholds let transients drive the creation of permanent floating Gaussians. We will sweep the threshold on the strongest baseline and report its best operating point at each noise level, so our method is compared against a tuned rather than a default filter.
Proposed Experiments
Experiments use the RobustNeRF benchmark and NeRF On-the-go, both of which provide clean held-out views, with synthetic per-view pose noise at 0°, 0.1°, 0.25°, 0.5°, 1°, and 2° of rotation, plus proportional translation noise. A real-jitter check uses NOTR driving video with its estimated poses.
Evaluation reports PSNR, SSIM, and LPIPS on clean test views. Test-view poses are aligned by a fixed-budget photometric pose fit applied identically to every method, so that no method is rewarded for how it handles the test cameras. We also report the recovered rotation error against the clean poses. To measure rare-structure retention, we report PSNR restricted to static regions visible in three or fewer training views. On scenes where distractor regions can be derived from the paired clean captures, we score static-versus-transient label precision and recall.
The decisive check comes at 0.5° and 1°. There, our method should hold within roughly 1 dB of the clean-pose oracle, while RobustSplat with learnable poses should show both lower PSNR and erased rare structure. The trimmed-loss ablation should reproduce that gap. If it does not, the starvation hypothesis is wrong.
A second check is the zero-noise row. The method must not hurt quality when poses are already clean.
The distractor-size sweep reports where attribution breaks down, defined as the point where a large rigid distractor starts to be absorbed into the pose correction.
References
- Calibration-Free BEV Representation for Infrastructure Perception
- RobustNeRF: Ignoring Distractors with Robust Losses
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- UPGS: Unified Pose-aware Gaussian Splatting for Dynamic Scene Deblurring
- Free-DyGS: Camera-Pose-Free Scene Reconstruction based on Gaussian Splatting for Dynamic Surgical Videos
- VDNeRF: Vision-only Dynamic Neural Radiance Field for Urban Scenes
- Back on Track: Bundle Adjustment for Dynamic Scene Reconstruction
- No Parameters, No Problem: 3D Gaussian Splatting without Camera Intrinsics and Extrinsics