Fast 3D reconstruction models can turn a handful of photos into a 3D scene of colored, semi-transparent blobs in under a second. These are 3D Gaussian splats, small ellipsoids whose position, shape, opacity and color are fit so that rendering them reproduces the photos. The speed comes at a cost: one-shot predictions blur detail and break on surfaces the training set did not cover. The slow alternative fits the blobs to each scene by gradient descent for thousands of steps. With few input views it also overfits the photos it can see. We propose a middle path. The network should not predict the final scene. It should predict a starting scene together with a per-parameter "how much to trust me" weight. A short optimizer then finishes the job at test time, minimizing photo error plus a penalty for straying from the prediction in proportion to that trust. The network is trained through the optimizer's answer using implicit differentiation, which means differentiating the condition that the optimizer has converged rather than replaying its steps. Training memory therefore does not depend on how many steps the optimizer runs.
The opportunity is that each of three research threads stops just short of this. Feed-forward reconstructors such as GS-LRM and DepthSplat remove per-scene optimization entirely. ReSplat adds back a four-step learned refinement, but that refinement minimizes no explicit objective and must be trained by backpropagating through all its steps. The differentiable-optimization literature knows how to differentiate through a solver's optimality conditions, from OptNet for quadratic programs to BPnP for camera pose. It has never been applied to the photometric objective of a radiance field, partly because the textbook recipe inverts a Hessian whose cost grows cubically in the number of variables, as Wang et al. emphasize.
Core Claim
Write the network's outputs as an initial Gaussian parameter vector $y_0$ and a positive diagonal trust matrix $\Lambda$, both functions of the input images and network weights $\theta$. The test-time layer solves
$$y^*(\theta) = \arg\min_y \; L_{\text{photo}}(y) + \tfrac{1}{2}(y - y_0)^\top \Lambda (y - y_0),$$
where $L_{\text{photo}}$ is the rendering error on the input views only. Training minimizes the novel-view rendering error of $y^*$, with gradients obtained from the stationarity condition by the implicit function theorem:
$$\frac{\partial y^*}{\partial (y_0,\Lambda)} = -\big(H_{\text{GN}} + \Lambda\big)^{-1}\frac{\partial g}{\partial (y_0,\Lambda)}.$$
Here $g$ is the gradient of the objective above, and $H_{\text{GN}} = J^\top J$ is the Gauss–Newton approximation to the photometric Hessian, with $J$ the Jacobian of rendered pixels with respect to Gaussian parameters.
We claim three things. First, the trust term is not cosmetic. It is what makes the inner problem well posed under sparse views. Gaussians that no input view constrains create flat directions, and those flat directions are exactly the singular-Hessian failure that OptNet and Gurumurthy et al. warn about. Adding $\Lambda$ guarantees a positive-definite system. Second, the cubic cost is avoidable. We never form the system matrix. We solve the linear system with conjugate gradient, an iterative method that only needs matrix–vector products. Each product $J^\top J v$ costs about one forward-mode and one reverse-mode pass through the rasterizer, and $\Lambda$ itself serves as the preconditioner. The backward pass therefore costs roughly as many render passes as conjugate-gradient iterations, regardless of how many solver steps the forward pass took. Third, and most important, $\Lambda$ can only be learned with a correct sensitivity. If the solver is treated as an identity map in the backward pass, the gradient with respect to $\Lambda$ vanishes. That is why the implicit gradient, rather than a cheap approximation, is the core ingredient.
Research Program
We build on a DepthSplat-style backbone with the compact Gaussian count used by ReSplat (246K Gaussians rather than MVSplat's 3.9M, per ReSplat). To this backbone we add a head that emits $\Lambda$ per Gaussian and per parameter group: position, scale, rotation, opacity and color. The forward solver is damped Gauss–Newton (also called Levenberg–Marquardt) using the same matrix-free conjugate-gradient machinery as the backward pass. It runs until the stationarity residual $\|g\|$ falls below a tolerance or a step cap is reached.
The analysis has three parts. First, we measure gradient bias from inexact forward solves. The bilevel survey flags this risk, so we log the residual at every step and correlate it with the error of the implicit gradient against finite differences on small scenes. Second, we study conditioning by tracking the spectrum of $H_{\text{GN}} + \Lambda$ through training. Third, we test the outlier-sensitivity tension between BPnP and Gurumurthy et al.. BPnP reports that least-squares sensitivity helps training. The bundle-adjustment study reports that outliers flood the gradient unless confidence weights gate them. Our $\Lambda$ plays the role of those confidence weights, so we will check whether it learns to down-weight Gaussians the input views cannot pin down. We will also check whether a robust photometric loss is still needed. Joint refinement of camera poses inside the same solve is a natural extension, since poses are just more variables in $y$. We treat it as secondary.
Baselines
The naive reference is DepthSplat run as published. It is a single forward pass with no refinement, and on DL3DV it reports a nearly 2 dB PSNR gain over MVSplat (DepthSplat). We retrain it on the same DL3DV-10K split we use, so it doubles as our initialization network.
The strongest existing method is ReSplat. It reports outperforming DepthSplat by 3.5 dB PSNR on DL3DV 8-view synthesis and 5 dB over single-pass versions at out-of-distribution resolutions. No absolute PSNR value for ReSplat on DL3DV or RealEstate10K was found on Lacuna, so we will re-run its released configuration under our data and GPU-hour budget. ReSplat counts as beaten only if we match it in-distribution within noise and exceed it on the out-of-distribution resolution and cross-dataset tests. Alternatively, we must match its quality at clearly lower training memory.
The classical reference is per-scene 3DGS optimization (3D Gaussian Splatting) initialized from the feed-forward prediction and run for 1K, 4K and 30K steps. ReSplat reports surpassing 3DGS at 4K iterations in four steps.
Two ablations isolate the key ingredient. The first trains the identical solver and $\Lambda$ head by unrolling $K$ solver steps, with $K$ chosen to match our training memory. The second uses an identity-Jacobian ("Jacobian-free") backward pass that treats $y^*$ as if it were $y_0$. A third ablation replaces the learned $\Lambda$ with a single tuned scalar. All methods share the backbone, training data and test-view protocol, and each receives an equal hyperparameter-search budget.
Why not just…?
Why not just run ordinary 3DGS optimization starting from the feed-forward prediction? This uses no learned prior during optimization. With few views, minimizing input-view error drifts away from good novel views, which is why ReSplat's four learned steps beat 4K steps of 3DGS. Our inner problem is the same optimization with a learned anchor, so the test-time fine-tuning baseline directly measures what the trust term buys.
Why not just unroll a few solver steps and backpropagate? This is a real threat. At $K \le 4$, unrolling may match us in-distribution. The case for implicit gradients rests on two measurable claims. Memory stays flat as the solver runs longer. A model trained this way also keeps improving when given more test-time steps than it saw in training, whereas unrolled models are tuned to their horizon. If the memory-matched unrolled ablation equals us on both counts, the implicit machinery is not justified, and we will report that.
Why not just use the cheap identity-Jacobian gradient? This is the most serious threat because it is nearly free. Its weakness is structural: it cannot learn $\Lambda$. If the scalar-$\Lambda$ ablation performs as well as the learned one, the argument for exact sensitivities collapses. That makes the comparison among learned-$\Lambda$-implicit, scalar-$\Lambda$ and Jacobian-free runs the decisive experiment.
Why not just use ReSplat's learned recurrent update? ReSplat's update is fast and strong. It minimizes no objective, however, and its own authors flag its k-nearest-neighbor attention as a scaling bottleneck on large scenes. Our update is the gradient-based minimizer of an explicit objective, so its residual is a convergence check we can actually compute. The out-of-distribution resolution and zero-shot ACID and ScanNet tests decide whether this matters in practice.
Proposed Experiments
We train on DL3DV-10K (DL3DV-10K) and evaluate on DL3DV 8-view synthesis and on RealEstate10K. Zero-shot transfer is tested on ACID and ScanNet, and out-of-distribution input resolutions follow ReSplat's protocol. Metrics are PSNR, SSIM and LPIPS on held-out views, reconstruction wall-clock, peak training memory, and the final stationarity residual.
Three checks decide the proposal. The first plots novel-view PSNR against the number of test-time solver steps, running out to several times the training horizon, for our model, the unrolled model and ReSplat. We predict ours rises monotonically and the others plateau or degrade. The second compares learned $\Lambda$, scalar $\Lambda$ and the Jacobian-free gradient at equal compute. A gap of at least 1 dB PSNR in favor of learned-$\Lambda$-implicit on out-of-distribution splits would support the core claim, and no gap would refute it. The third plots peak training memory against solver steps, which should be flat for implicit gradients and linear for unrolling. Gradient correctness is verified against finite differences on small scenes before any of these runs.
References
- GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
- DepthSplat: Connecting Gaussian Splatting and Depth
- ReSplat: Learning Recurrent Gaussian Splats
- OptNet: Differentiable Optimization as a Layer in Neural Networks
- End-to-End Learnable Geometric Vision by Backpropagating PnP Optimization
- Automatically Learning Compact Quality-aware Surrogates for Optimization Problems
- From Variance to Veracity: Unbundling and Mitigating Gradient Variance in Differentiable Bundle Adjustment Layers
- Investigating Bi-Level Optimization for Learning and Vision From a Unified Perspective: A Survey and Beyond
- 3D Gaussian Splatting for Real-Time Radiance Field Rendering
- DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision