Feed-forward 3D reconstructors are networks that turn a few photos into a full 3D scene in under a second. Systems like GS-LRM, DepthSplat and ReSplat output a cloud of small colored 3D blobs (3D Gaussian Splatting). They are fast only because they skip the minutes-to-hours of per-scene gradient descent that classical reconstruction needs. All three assume the camera positions and orientations of the input photos (the "poses") are known exactly. When the poses are wrong, the network places each photo's blobs in the wrong spot, and the scene comes apart. We propose to repair the poses at test time without retraining the network and without backpropagating through it. We treat the frozen reconstructor as a black box that scores a candidate set of poses. We then search over the handful of pose parameters with a zeroth-order optimizer, meaning one that estimates gradients only from differences in the score under small random pose nudges. A running average of past estimates (momentum) filters out the noise of those estimates.
The idea in two sentences: a scene built from slightly wrong poses disagrees with itself, because the blobs built from photo A, rendered into camera B, do not match photo B. We minimize that disagreement over the camera poses using only forward passes of the frozen network. A deliberately large and then shrinking nudge size blurs the disagreement landscape early on, so the search can escape the false alignments that trap ordinary gradient-based pose refinement.
Core Claim
Pose search has few dimensions: six per non-reference camera, so 6 for two views and 42 for eight. That is exactly the setting where forward-only gradient estimation is cheap. The dimension penalty that ZO-AdaMM proves (its convergence rate degrades from $O(\sqrt{d}/\sqrt{T})$ to $O(d/\sqrt{T})$ relative to first-order methods) costs very little when $d$ is under 50. Our claim is that, with a fixed budget of roughly 64 to 256 forward passes, a momentum-filtered two-point estimator with an annealed perturbation radius recovers poses from initial errors several times larger than first-order pose refinement can handle. We expect it to match first-order accuracy near the solution, at the memory cost of inference alone.
The mechanism is the standard smoothing identity. The objective the zeroth-order method actually descends is
$$f_\mu(\theta) = \mathbb{E}_{u \sim \mathcal{N}(0,I)}\big[f(\theta + \mu u)\big],$$
where $\theta$ is the stacked pose vector, $f$ is the cross-view disagreement of the frozen reconstructor, and $\mu$ is the perturbation radius. This is $f$ convolved with a Gaussian of width $\mu$. A large $\mu$ erases the narrow false minima. Shrinking $\mu$ over the steps gives a coarse-to-fine search for free. RAIN-GS (Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting) motivates this. It shows that random-start 3DGS fails because pixel gradients fit high frequencies too early. It fixes this with a hand-scheduled low-pass rendering filter, reaching 27.23 PSNR on Mip-NeRF 360 from 10 random points against 22.19 for naive random initialization. Here the low-pass behavior comes from the estimator itself and acts on poses rather than on blobs.
Research Program
The objective. Rendering a view's own blobs back into its own camera reproduces that photo almost regardless of pose, so self-reconstruction is useless as a score. We instead define $f(\theta)$ as cross-view disagreement. We run the frozen reconstructor once with poses $\theta$. Then, for each camera $i$, we render only the blobs originating from the other views into camera $i$ and compare against photo $i$. The comparison combines a pixel term and a deep-feature term, following ReSplat's use of pixel- and feature-space error maps. The feature term is there to tolerate the lighting changes that break correspondence matching in iGaussian. The first camera is fixed to remove the global gauge freedom. Pixels with no cross-view coverage are masked out, as are pixels where the rendered opacity is low. This objective works for two views, which is the hardest and most common sparse setting.
The optimizer. Each step draws a random direction and evaluates the objective at $\theta \pm \mu u$. The two evaluations are batched in a single forward call, since no activations need storing. The step then forms the directional estimate and updates first and second moments in the R-AdaZO way (Refining Adaptive Zeroth-Order Optimization at Ease): the per-coordinate scale is computed from the smoothed first moment rather than the raw noisy estimate. Per-coordinate scaling matters here more than in prior zeroth-order applications, because rotation (radians) and translation (scene units) have very different sensitivities. A shared learning rate is exactly the anisotropy problem adaptive methods address. Directional-derivative clipping, as introduced in QZO, guards against the large score spikes that occur when a nudge pushes a camera into a degenerate configuration.
The radius schedule. $\mu$ starts at a rotation scale comparable to the expected initial error and decays geometrically to a small floor. The schedule is tuned once on a training split and never per scene. We will also study a variant that sets $\mu$ from the spread of the momentum buffer, to address the open problem of automating decay parameters noted in R-AdaZO and in SMTP.
Analysis. We will map the landscape directly. For two-view scenes we sweep one rotation axis and plot $f$ and $f_\mu$ for several $\mu$. This tests whether the false minima are narrow, so that smoothing removes them, or wide, so that smoothing just shifts them. That question decides whether the core claim can hold.
Relative to prior work, the closest relative is Feature-Metric Registration. It also drives a rigid transform with finite differences over a learned, smooth feature space and reaches 80° tolerance on ModelNet40. However, it uses raw finite differences with no momentum or radius annealing, and it aligns isolated point clouds rather than refining camera poses through a scene reconstructor. PF3plat solves pose-free splatting with pretrained depth and matching networks plus learned refinement, entirely feed-forward. We take its output as one starting point and add a test-time repair stage it lacks. iGaussian avoids render-compare-refine loops for localization against a prebuilt scene, reporting that iterative methods drop to 0% success in low overlap. That result motivates our search for a wider basin, but iGaussian presupposes an already optimized scene, whereas we reconstruct and pose jointly.
Baselines
The naive reference point runs the frozen reconstructor (DepthSplat for two and eight views, ReSplat where weights are available) on the noisy poses with no repair. This measures how much damage pose error causes and bounds the gain from repair from below. The matching upper reference runs the same network on ground-truth poses.
The strongest existing method is PF3plat. It reports 23.59 dB PSNR on RealEstate10K two-view synthesis in 0.39 seconds, a 4.0 dB improvement over CoPoNeRF, and more than 21 dB zero-shot on DL3DV (PF3plat). We will run its released model on the identical RealEstate10K and DL3DV test splits and evaluate its poses with our metrics. We will also use its estimated poses as an initialization for our repair, so the question becomes whether repair adds value on top of the best feed-forward poses. Beating it means higher novel-view PSNR and lower relative rotation error on the same scenes, with total time (PF3plat plus repair) reported honestly. A small PSNR gain that only appears in extrapolated views, where PF3plat admits weakness, would still count, provided it is reported as such.
The most direct competitor on the mechanism is first-order pose refinement. We minimize the same cross-view objective with Adam, backpropagating through the frozen reconstructor and renderer into the poses. It gets the same initial poses, the same wall-clock budget, the same number of hyperparameter trials on the training split, and the same coarse-to-fine variant (a blurred-rendering schedule) so that annealing is not unique to our method. We report peak memory for both. Beating it means a wider capture range, measured as success rate at large initial errors, at equal accuracy for small errors.
As a per-scene optimization reference without poses, we cite InstantSplat's roughly 53 seconds per scene as reported by PF3plat. If our repair loop cannot finish well under that, the overhead argument fails.
Ablations isolate the ingredients. ZO-SGD with a fixed radius removes momentum and annealing. Momentum with a fixed radius isolates annealing. ZO-AdaMM versus R-AdaZO isolates the second-moment fix. Removing the feature term tests lighting robustness.
Why not just…?
Why not just backpropagate through the differentiable renderer and network into the poses? This is the real threat, and the first-order baseline above is designed to settle it. For two views at modest resolution, memory is not prohibitive on an A100, so the case for zeroth-order rests on capture range and on many-view, high-resolution or edge settings. The evidence for brittleness is indirect: iGaussian reports iterative photometric refinement collapsing to 0% success in low-overlap cases. That result concerns localization against a fixed scene, not our objective. If first-order refinement with a matched blur schedule reaches the same capture range, we will report that the advantage is memory alone and restrict the claim to the eight-view, 512×960 regime, where DepthSplat-scale activations dominate.
Why not just use a population-based derivative-free optimizer such as CMA-ES or multi-start random search? In 6 to 42 dimensions this is a strong alternative, since CMA-ES also smooths implicitly and adapts scale. We run it as a co-equal baseline at identical forward-pass budgets. Our prediction rests on budget arithmetic. A two-point step costs 2 evaluations. A CMA-ES generation in 12 dimensions uses around a dozen. So under 64 to 128 passes, momentum zeroth-order takes several times more updates. If CMA-ES matches or wins, the contribution narrows to the cross-view objective and the annealing schedule, and we will say so.
Why not just rely on feed-forward pose estimation like PF3plat and accept its error? PF3plat already beats prior pose-free methods by 4 dB. The PF3plat-initialized experiment answers this directly. If repair yields no gain on its poses at realistic noise levels, the method is only useful for larger corruptions, such as wrong IMU priors or drifted SLAM poses, and we will scope it to those.
Proposed Experiments
We use RealEstate10K and DL3DV, the benchmarks shared by PF3plat, DepthSplat and ReSplat, with ACID as a held-out domain. Initial poses come from two sources. The first is ground truth corrupted by controlled rotation noise from 0° to 30° plus proportional translation noise. The second is PF3plat's estimates. We report relative rotation and translation error, novel-view PSNR, SSIM and LPIPS separated into interpolated and extrapolated targets, forward passes used, wall-clock time and peak GPU memory.
The decisive check is a capture-range curve: success rate (final rotation error below 2°) against initial error, for the naive reference, first-order refinement, CMA-ES, and our method. The claim holds if our curve stays high where first-order collapses, at equal budget. It fails if the curves coincide. A second check is the landscape sweep, which should show smoothing removing minima rather than displacing them. A third check is scaling from two to eight views, where the expected growth of required passes with $d$ is roughly linear, following the ZO-AdaMM bound. Memory should stay flat for our method and grow for backpropagation. Finally, we run a lighting-shift subset of DL3DV, comparing pixel-only and pixel-plus-feature objectives. This tests whether forward-only repair survives the conditions that break correspondence pipelines.
References
- GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
- DepthSplat: Connecting Gaussian Splatting and Depth
- ReSplat: Learning Recurrent Gaussian Splats
- ZO-AdaMM: Zeroth-Order Adaptive Momentum Method for Black-Box Optimization
- Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting
- iGaussian: Real-Time Camera Pose Estimation via Feed-Forward 3D Gaussian Splatting Inversion
- Refining Adaptive Zeroth-Order Optimization at Ease
- Fine-tuning Quantized Neural Networks with Zeroth-order Optimization
- A Stochastic Derivative Free Optimization Method with Momentum
- Feature-Metric Registration: A Fast Semi-Supervised Approach for Robust Point Cloud Registration Without Correspondences
- PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting