Turning a handful of photos into a 3D scene still usually means optimizing that scene from scratch. Methods such as 3D Gaussian Splatting, which represents a scene as hundreds of thousands of small colored blobs, run thousands of gradient steps per scene. Feed-forward reconstruction networks skip that loop by predicting the blobs directly. The best of them, such as ReSplat, add a few learned "correct the rendering error" iterations, and those iterations are where most of their quality advantage comes from.
This note proposes to make those correction steps principled. We borrow one idea from quantum singular value transformation (QSVT), a quantum-algorithms toolkit that applies an arbitrary polynomial to the singular values of a matrix without ever decomposing it. Every refinement step, learned or hand-designed, is some function of the renderer's sensitivity matrix applied to the rendering error. QSVT theory tells us which polynomials are worth computing, how high their degree must be, and how to evaluate them stably. Concretely, a network predicts, per scene, a small set of Chebyshev coefficients defining a spectral filter. The filter is applied through a few Jacobian-vector products of the renderer. The aim is the quality of short test-time optimization at the cost of a feed-forward pass. No quantum hardware is involved; the contribution is to carry QSVT's mathematics into a regime where classical computation already wins.
Core Claim
Let $J$ be the Jacobian of the rendered input views with respect to all Gaussian parameters, and $r$ the photometric residual on those views. A Gauss-Newton step solves $(J^\top J)\Delta = J^\top r$. Gradient descent, conjugate gradient, and Chebyshev iteration all return an update of the form
$$\Delta = f(J^\top J)\, J^\top r, \qquad f(x) = \sum_{k=0}^{K} c_k\, T_k(x/\lambda_{\max}),$$
where $T_k$ is the $k$-th Chebyshev polynomial, $\lambda_{\max}$ is an estimate of the largest eigenvalue of $J^\top J$, and the $c_k$ are coefficients. Computing $\Delta$ costs $K$ matrix-free products with $J^\top J$, each roughly one render plus one backward pass. QSVT works in exactly this "polynomial of the spectrum" picture. The block-encoded circuits of Lopatnikova et al. realize degree-$d$ polynomials $f(\sigma)$ on singular values.
Our claim has three parts. First, with sparse views the right $f$ is not $1/x$. Exact inversion fits the input views and amplifies directions the views barely constrain, which are the floaters behind surfaces that feed-forward models are known to produce. What we want is a regularized inverse: invert the well-observed spectrum and suppress the rest, like the singular value thresholding filters QSVT implements. Second, the best filter depends on the scene (texture, baseline, depth range), so it should be predicted from scene features rather than fixed. Third, a degree-$K$ learned filter trained against held-out-view quality will beat equal-cost conjugate gradient, gradient descent, and black-box learned recurrence.
What is new is not polynomial preconditioning itself. It is (a) treating feed-forward 3D refinement as learned spectral filtering of the rendering Jacobian, with filter shape trained for novel-view quality, and (b) a measurement of whether existing learned refiners implicitly behave like such filters.
Relation to Prior Work
The feed-forward side supplies the starting point and the competitors. GS-LRM regresses one Gaussian per pixel in a single pass in about 0.23 s. DepthSplat predicts Gaussian centers from fused stereo and monocular depth. ReSplat adds four recurrent updates driven by rendering-error maps, without gradient backpropagation, and gains 3.5 dB PSNR over DepthSplat on DL3DV 8-view. ReSplat is the closest existing work. Its update is an unconstrained attention-and-kNN network acting on error maps; ours is a structured operator whose only learned part is a short coefficient vector. That gives an interpretable knob (degree, spectral shape) and a direct comparison to optimization. Score Jacobian Chaining motivates the problem by documenting the minutes-to-hours cost of per-scene optimization.
The QSVT side supplies the method and the constraints. Orsucci and Dunjko's positive-definite linear-system work designs Chebyshev-based polynomials for $1/x$. It gets $O(\sqrt{\kappa})$ dependence on the condition number $\kappa$ for restricted positive-definite classes, and also proves an $\Omega(\kappa)$ lower bound in general. The classical analogue, Chebyshev acceleration, needs degree on the order of $\sqrt{\kappa}\log(1/\epsilon)$ for a residual polynomial to fall below $\epsilon$ on a spectrum of condition number $\kappa$. We use this to predict how large $K$ must be once we measure $\kappa$ of the regularized $J^\top J$. The lower bound warns that if the relevant spectrum is badly conditioned, no cheap polynomial will suffice, which is exactly why we aim to filter rather than invert. The improved classical singular value transformation paper supplies the evaluation method: Clenshaw recurrence for matrix Chebyshev series, with a stability analysis showing that errors largely cancel. We rely on that for half-precision rendering. The same paper, together with the data-loading caveat in the matrix geometric means work, is why we make no quantum-speedup claim. Loading a multi-million-parameter Jacobian into a block-encoding would cost as much as the classical computation.
The two existing bridges between these areas point elsewhere. QNeRF replaces the NeRF network with a variational quantum circuit, still optimized per scene at up to 50 hours. It is a cautionary example of quantum methods worsening per-scene cost, and we do not follow it. SoftDecay applies a learned singular value transformation to transformer hidden states via an explicit SVD, at about 1.5x cost. We share its view that reshaping a spectrum is a useful learned operation, but we act on the rendering Jacobian, matrix-free, without any SVD.
Research Program
The first stage is diagnostic and could change the project. On 50 DL3DV and RealEstate10K scenes, we initialize from DepthSplat and use Lanczos iteration through the differentiable rasterizer to estimate the spectrum of $J^\top J$ restricted to the Gaussians. We then ask two questions. How spread is the spectrum, and how much of the held-out-view error lies in its poorly observed tail? And does ReSplat's update lie close to a low-degree Krylov subspace, meaning the span of $J^\top r, (J^\top J)J^\top r, \dots$? We answer the second by regressing the observed ReSplat update onto that basis. If ReSplat's updates are already well explained by a degree-4 polynomial, our contribution narrows to making that structure explicit and cheaper. If they are not, the comparison between the two update families becomes the central question.
The second stage builds the module. A light head reads pooled scene tokens from the DepthSplat backbone and outputs, at each outer iteration, $\lambda_{\max}$ scaling plus coefficients $c_0,\dots,c_K$, with $K$ between 4 and 16. The update uses Clenshaw recurrence with Jacobian-vector and vector-Jacobian products from the 3DGS rasterizer. We run two to four outer Gauss-Newton relinearizations. Densification and pruning are excluded so the parameter set stays fixed. Training is end to end on DL3DV-10K and RealEstate10K, with the loss computed only on held-out target views, so the filter learns to trade input-view fit against generalization. To control memory we backpropagate through the recurrence with gradient checkpointing, and we report truncated backpropagation as a variant.
The third stage is analysis. We plot learned filter shapes $f(x)$ against Tikhonov and truncated-inverse references. We correlate filter shape with view count and baseline, and we test whether measured $\kappa$ predicts the degree at which held-out PSNR saturates, as the $\sqrt{\kappa}$ argument suggests.
Baselines
The naive reference is the DepthSplat initialization with no refinement. This is the same backbone our module refines, so any gain is attributable to the update rule. Above it sits the strongest existing feed-forward method, ReSplat, run from released code or retrained on the same DL3DV-10K and RealEstate10K splits, view counts, and resolution as our model. For it we report only numbers we reproduce. The file gives ReSplat's 3.5 dB PSNR margin over DepthSplat on DL3DV 8-view but no absolute PSNR (no reported result found on Lacuna for ReSplat's absolute DL3DV 8-view PSNR). Beating ReSplat means higher held-out PSNR and SSIM at no greater wall-clock and memory, or equal quality at clearly lower cost.
The optimization baselines, all starting from the same DepthSplat initialization, are the following. First, Adam test-time fine-tuning of the Gaussians for a matched number of render-plus-backward passes. Second, Gauss-Newton with conjugate gradient, with iteration count and damping tuned on validation scenes. Third, Chebyshev iteration with fixed, spectrum-derived coefficients. CG gets the same validation tuning budget we give our hyperparameters. Every method is charged in renderer passes and in A100 wall-clock. As an upper reference that pays the full per-scene price, dense per-scene optimization on DL3DV-140 reaches 31.22 PSNR with Zip-NeRF, as reported by DL3DV-10K. Note that this figure uses dense views, so it bounds rather than competes.
The key ablations isolate the two ingredients. Coefficients learned once for all scenes, versus predicted per scene, tests whether adaptivity matters. Training the filter on input-view loss, versus held-out-view loss, tests whether the gain comes from learning a regularizing filter shape rather than a faster inverse.
Why not just…?
Why not just run a few steps of ordinary test-time optimization after the feed-forward pass? This is the simplest competitor, and at large step counts it will win on quality. It is the per-scene loop we are trying to avoid. The question is the low-budget regime of 4 to 16 renderer passes. There, plain gradient steps apply the degree-$K$ polynomial $\sum_k (1-\eta x)^k$, which converges poorly on spread spectra. Our matched-pass comparison measures this directly, and a loss to Adam at equal passes would falsify the proposal.
Why not just use conjugate gradient, which already finds the best polynomial adaptively? This is the most serious threat. CG picks the polynomial that minimizes the error in the $J^\top J$-weighted norm for the input-view least-squares problem, and with early stopping it already regularizes. Our bet is that the optimal input-view fit is not the optimal novel-view result under sparse views, and that a filter trained on held-out views can do better. A secondary advantage is that Chebyshev/Clenshaw needs no inner products, so it is easier to train through and more stable at low precision. If tuned, early-stopped CG matches the learned filter, we will report it. The project then reduces to the claim that feed-forward models should end with a CG polish, which is still a useful negative result.
Why not just run more ReSplat iterations? ReSplat already reports large out-of-distribution gains from recurrence, so more iterations is natural. But each iteration's global attention and kNN cost grows with primitive count, which the research file lists as an open bottleneck. Its unconstrained update also offers no convergence guarantee as $T$ grows. We sweep ReSplat's $T$ against our $K$ on matched wall-clock and test which curve saturates first and which degrades on held-out resolutions.
Proposed Experiments
The main evaluation is DL3DV 8-view and 12-view at 512×960, plus RealEstate10K two-view. We report PSNR, SSIM, and LPIPS on held-out views, along with renderer passes, wall-clock, and peak memory. Generalization is tested zero-shot on ScanNet and ACID and at out-of-distribution resolutions, where ReSplat reports its largest recurrent gains.
Three checks decide the outcome. The first is the quality-versus-compute curve. The learned filter must lie strictly above both tuned CG and Adam at every budget from 4 to 32 renderer passes, and match or beat ReSplat at equal wall-clock. The second is the per-scene versus global coefficient ablation, together with the held-out versus input-view loss ablation. If neither matters, the spectral-filtering thesis is wrong even if the numbers look good. The third is the conditioning check: the measured $\kappa$ of the damped $J^\top J$ should predict the degree at which quality saturates, roughly as $\sqrt{\kappa}$. If saturation is unrelated to $\kappa$, the QSVT-derived reasoning is not what explains the method, and we will say so.
References
- An introduction to quantum computing for statisticians and data scientists
- GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting
- DepthSplat: Connecting Gaussian Splatting and Depth
- ReSplat: Learning Recurrent Gaussian Splats
- Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation
- On solving classes of positive-definite quantum linear systems with quadratically improved runtime in the condition number
- An Improved Classical Singular Value Transformation for Quantum Machine Learning
- Quantum algorithms for matrix geometric means
- QNeRF: Neural Radiance Fields on a Simulated Gate-Based Quantum Computer
- Addressing token uniformity in transformers via singular value transformation
- DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision