Pose Syndromes: Telling Camera Error from Moving Distractors in Unposed 3D Reconstruction

When a 3D scene is reconstructed from photos, two kinds of error look alike in the loss. If a camera's estimated position is slightly wrong, every pixel of its rendered image disagrees a little with the photo. If a pedestrian walked through the shot, a small patch of pixels disagrees a lot. Current methods handle these problems separately. Pose-refinement methods let the optimizer correct camera positions, but they assume nothing in the scene moves. Distractor-removal methods discard the worst-matching pixels, but they assume camera positions are already correct. Run both at once on casual, unposed footage and they fail together. While the cameras are still misaligned, every pixel has a large error, so "drop the worst 50% of pixels" cannot tell a misaligned camera from a person walking by.

We propose a fix borrowed from quantum error correction. There, a syndrome is a measurement built to reveal one type of error while staying blind to others. In noise where one error type is far more common than the other, codes devote unequal amounts of redundancy to the two types. Our vision analog is simple. For each image, we first remove the part of the error that any small camera correction could explain: the projection of the residual onto that camera's six motion directions. Only what remains is used to decide which pixels belong to distractors and where the 3D model should grow new geometry. Camera updates are made robust by requiring agreement across the whole image. Distractor labels require that the error survive pose correction and be spatially coherent. The claim is that this pose-explained residual lets camera refinement and distractor rejection run jointly from poor initial poses, a setting in which, according to the research file, no existing method has been demonstrated.

Core Claim

Three lines of work frame the gap. BARF treats camera poses as learnable and backpropagates the photometric loss into them, with low-frequency inputs enabled first to widen the basin of attraction. It recovers synthetic poses from an average 14.9° rotation error to under 0.2°, but it assumes a rigid, static world. RobustNeRF, NeRF On-the-go, RobustSplat and 3DGS-HPC reject distractors with three tools:

  • trimmed losses,
  • learned uncertainty,
  • delayed growth of Gaussians plus patch classifiers.

All four take poses precomputed by structure-from-motion as fixed. The papers that already join pose refinement with dynamics, BASED and SplineGS, penalize all photometric residuals the same way. BASED leans on external dense correspondences to separate camera motion from tissue motion.

Our claim is narrow and testable. The specific reason robust losses break under pose error is that they rank raw residuals. Ranking residuals after removing the best robust six-degree-of-freedom correction per image restores their discriminative power. The method needs no semantic masks and no correspondence network, and it is cheap, because a 6×6 solve per image is negligible next to rendering.

The Mechanism

Let $r_i$ be the per-pixel photometric residual of image $i$ under the current scene and pose. Let $J_i$ be the analytic Jacobian of the rendered image with respect to a small pose change $\delta \in \mathbb{R}^6$. Schmidt et al. show these camera gradients are cheap in Gaussian splatting, the rasterized point-primitive representation we build on. Let $W_i$ be a diagonal matrix of robust per-pixel weights. The pose syndrome step computes

$$\delta_i^\star = \arg\min_{\delta} \big\| W_i^{1/2}(r_i - J_i \delta) \big\|^2, \qquad e_i = r_i - J_i \delta_i^\star ,$$

where $e_i$ is the pose-unexplained residual. Because $J_i$ has only six columns, a localized distractor barely moves $\delta_i^\star$ once the robust weights down-weight it. A global misalignment, in contrast, is absorbed almost entirely into $J_i\delta_i^\star$. The weights are refit for a few iterations (iteratively reweighted least squares) from $e_i$ rather than from $r_i$.

Each signal then has its own job. Camera updates use $\delta_i^\star$ together with ordinary gradients, under BARF-style coarse-to-fine scheduling. The weights that mark pixels as distractors are computed from $|e_i|$, smoothed over 16×16 patches as in RobustNeRF and 3DGS-HPC. The gradient that triggers adaptive densification, the step where splatting clones or splits Gaussians, is computed from $e_i$ only. This targets the failure RobustSplat documents: densification treats any large error as missing detail. Under joint pose optimization, that failure gains a second source, because pose error also spawns misplaced Gaussians.

The quantum error correction literature supplies the design principle rather than any code construction, and the borrowing has three parts.

  • Separate checks for separate error types. Ioffe and Mézard pair different classical code families for the two error types. Fan et al. give one error type a distance that grows linearly with block length, while the other gets high rate.
  • Turn the dominant error into a simpler problem. The ZXXZ surface code in Suppressing quantum errors by scaling a surface code logical qubit rotates its checks so that biased noise reduces to repetition-like chains.
  • Skew the checks toward the dominant error. Vallero et al. report about 10% better resilience to radiation bursts at no hardware cost by doing this.

In our setting, pose error is "dense but low-dimensional" and distractors are "sparse but unstructured." We give them separate checks, each blind to the other. We also require more redundancy before changing pose than before labeling a patch transient. This is an analogy, and we claim nothing quantum. Its value is in dictating the design: checks are constructed to be insensitive to the other error class, rather than being one shared robust loss.

A second, cheaper check targets the "rare object" failure that 3DGS-HPC describes, where static content seen in only one or two views is discarded as a distractor. A patch flagged in frame $i$ is re-rendered in the temporally adjacent frames where it projects. If its pose-unexplained residual is low there, it is restored as static. Transients do not persist in place across frames; rarely seen static objects do.

Research Program

We build on a standard Gaussian splatting codebase and add three pieces:

  • per-image pose parameters in the tangent space of rigid motions,
  • the syndrome solve,
  • the two gating rules (distractor weights and densification).

The first phase is a controlled synthetic benchmark. We render NeRF Synthetic scenes, composite moving objects into a known fraction of frames, and perturb poses following BARF's protocol. This gives ground-truth distractor masks and poses, so we can sweep distractor coverage against pose perturbation magnitude. We can then map exactly where raw-residual trimming collapses under joint refinement and where the syndrome version does not.

The second phase moves to real distractor benchmarks with perturbed or partly missing poses. The third phase, explicitly a stretch, applies the same projection inside SplineGS on NVIDIA and DAVIS monocular video. There the "pose-unexplained" residual must be further split between spline-modeled motion and transients. This tests whether the idea survives when the scene is genuinely dynamic rather than merely contaminated.

We also analyze when the method should fail, namely when the six pose directions are nearly degenerate:

  • textureless walls, where $J_i^\top W_i J_i$ is ill-conditioned;
  • a distractor large enough to dominate the frame.

We will report the condition number of $J_i^\top W_i J_i$ as a diagnostic and test whether it predicts failure.

Baselines

The do-nothing reference is plain Gaussian splatting trained with the perturbed poses held fixed. It shows how much damage pose error and distractors do together.

The pose-only reference is BARF-style coarse-to-fine pose refinement in the same splatting code with an L1 photometric loss and no distractor handling. It isolates what pose refinement achieves when distractors are present. Its clean-scene anchor is BARF's reported 27.50 dB PSNR with under 0.2° rotation error on NeRF Synthetic from 14.9° perturbation.

The strongest baseline, and the one a reviewer would expect, is the naive union: the same pose refinement combined with the best available distractor handling applied to raw residuals. Concretely, we run it twice, once with RobustSplat's delayed growth and mask bootstrapping and once with 3DGS-HPC's hybrid patch classifier. Every pipeline gets the same iteration count, the same GPU (an RTX 4090, matching 3DGS-HPC's reported 13–17 minute training regime), and an equal hyperparameter search budget per method on held-out scenes.

The upper-bound reference is RobustSplat and 3DGS-HPC run with the benchmark's provided poses. The file reports these comparisons:

Benchmark Method Reported result
RobustNeRF "Robotics Lab" RobustNeRF 30.87 dB PSNR (vs 25.22 dB for Mip-NeRF 360)
NeRF On-the-go "Patio" NeRF On-the-go 21.41 PSNR (vs 20.54 for RobustNeRF)
RobustNeRF benchmark 3DGS-HPC 1.39 to 5.54 dB above native splatting
On-the-go and RobustNeRF RobustSplat no per-scene number found on Lacuna

We will rerun all of these ourselves under our protocol.

Beating the naive union means three things at matched compute: lower median rotation and translation error, higher PSNR, and higher distractor-mask IoU on the synthetic suite. It also means closing most of the gap to the known-pose upper bound on real scenes. A secondary success criterion is staying within roughly half a decibel of that bound, a target we set, not a reported figure.

The key ablation removes the projection: identical code, with weights and densification computed from $r_i$ instead of $e_i$. If this ablation matches the full method, the core claim is false. Further ablations remove the densification gating and the temporal persistence check separately.

Why not just…?

Why not just add a Huber or other robust loss to BARF? A robust loss on raw residuals has the same weakness as trimming. Early in training, pose error makes nearly every residual large, so down-weighting the large ones either suppresses the signal pose refinement needs or keeps the distractors. The research file's account of RobustNeRF's trimmed loss is relevant here: it discards a fixed fraction of data and loses thin static structure even with correct poses. The Huber-weighted joint pipeline is included among our raw-residual baselines. The projection ablation compares it directly against the same weights computed after pose correction.

Why not just mask distractors with an off-the-shelf segmenter and then run rigid pose refinement? 3DGS-HPC reports that semantic masks miss non-object transients such as moving shadows and illumination changes. Split4D shows 2D tracking-based decomposition reaching 0.592 mIoU versus 0.893 for native 3D decomposition on Multi-Human, as reported in Split4D. Even so, a segmenter is a real competitor on scenes whose distractors are people. We will run a segmenter-masked pose-refinement baseline and report results broken down by distractor type, accepting a loss on person-only scenes if that is what the data show.

Why not just get poses from structure-from-motion with dynamic regions masked, then run RobustSplat? Where structure-from-motion succeeds, this is our upper-bound reference, not a competitor we need to beat. The research file notes it fails outright on sparse in-the-wild collections and monocular dynamic video, per SplineGS and MS-GS. BASED reports that holding even slightly noisy poses fixed caps surgical reconstruction at 26.97 dB, against 32.227 dB with joint optimization. Our contribution is the regime where this pipeline breaks, and we will report its failure rate scene by scene.

Why not just schedule it: refine poses first, then switch on distractor rejection? This is the most serious threat, because BARF's coarse-to-fine schedule already makes early residuals smoother. The weakness is that the first stage fits poses using every pixel, distractors included, so a large or persistent distractor biases the registration that the second stage inherits. We will implement this two-stage schedule with the same total budget and compare across the synthetic sweep. If it matches our method at all distractor fractions, the proposal reduces to a scheduling trick, and we would report that.

Proposed Experiments

The decisive experiment is the synthetic sweep. We expect a crossover: as distractor coverage grows, raw-residual trimming with joint pose refinement degrades sharply, while the syndrome version degrades gracefully. Ground-truth masks let us separate the two failure routes, poses dragged by distractors versus distractors absorbed as geometry.

On NeRF On-the-go and the RobustNeRF benchmark, we evaluate under pose perturbation and, for forward-facing scenes, from identity initialization, as BARF did on LLFF. We report three metrics:

  • rotation and translation error after similarity alignment to the provided poses,
  • PSNR, SSIM and LPIPS on test views with test-time pose alignment,
  • the number of Gaussians, to check whether gating densification on $e_i$ prevents the floater growth RobustSplat describes.

A rare-object subset, scenes with static items visible in few views, tests the temporal persistence check. A conditioning analysis on textureless scenes tests whether the method fails when the six pose directions become degenerate. The SplineGS extension on NVIDIA and DAVIS is reported as exploratory; the file gives no PSNR figure there, so we will establish the baseline ourselves.