Project Once, Refine Only When Needed: Low-Rank Tensor Certificates for Feed-Forward 3D Reconstruction

Feed-forward 3D reconstruction networks turn photos of a scene into 3D points in tens of milliseconds per image pair. Combining many views into one consistent scene still usually needs a slow iterative clean-up step. That step is either a gradient-descent "global alignment" over all pairs or a bundle adjustment, the classical nonlinear least-squares fit of cameras and points. Classical multi-view geometry holds an algebraic fact the networks ignore. If you stack the three-view relations of every camera triplet into one large "block trifocal tensor," the result has a fixed, tiny rank no matter how many cameras there are. A trifocal tensor is the 3×3×3 array encoding how a point seen in two images constrains its position in a third. Its rank being fixed means consistency across all views can be tested and enforced by one low-rank projection rather than a long optimization.

We propose to build this tensor from the network's own triplet predictions and project it once onto its known low-rank form to recover all cameras. The size of the projection residual, meaning how far the network's predictions sit from any geometrically possible set of cameras, then serves as a certificate. It decides whether iterative refinement can be skipped entirely, or run only on the views it flags. The payoff is a latency–accuracy curve that neither existing option gives. Fully feed-forward models are fast but unchecked. Iterative pipelines are accurate but pay their full cost on every scene, including the easy majority.

Core Claim

Our claim has two parts. First, a single truncated higher-order singular value decomposition, with no alternating or thresholding iterations, of a sparsely sampled block tensor built from neural triplet predictions recovers camera locations about as well as iterative global alignment. This should hold on scenes where the network is roughly right, at a small fraction of the time. Second, the normalized residual of that projection predicts reconstruction failure better than the network's own confidence scores. Using it to gate refinement reaches the accuracy of always refining at a much lower average latency. The second part is the one we expect to be new, and it is the one a reviewer should hold us to.

The structure we use comes from Tensor-Based Synchronization and the Low-Rankness of the Block Trifocal Tensor. That paper proves that the $3n \times 3n \times 3n$ block tensor $T^n$ for $n$ cameras has exact multilinear rank $(6,4,4)$ in the noise-free case. Its companion QuadSync proves that the four-view analogue $Q^n$ has rank $(4,4,4,4)$, with the stacked camera matrices as its factors. Both papers feed classical feature-based tensor estimates into iterative solvers: hard-thresholded decomposition in the first, and a scheme combining alternating-direction updates with reweighted least squares in the second. Neither uses learned inputs, and neither uses the residual as a decision signal. On the other side, MV-DUSt3R+ removes DUSt3R's iterative global alignment by letting attention learn multi-view consistency implicitly. That motivates our work, since it shows iteration can be dropped. It also leaves the gap we fill: nothing checks whether the implicit consistency actually holds.

Research Program

The pipeline has four stages. First, we sample camera triplets from the viewing graph. For quadruplets we would add scenes with near-straight-line camera motion, where QuadSync reports that pairwise and trifocal ranks collapse. Second, we run a multi-view network (MV-DUSt3R+, restricted to three or four input views) on each sampled tuple. It predicts that tuple's cameras in a shared frame, and we convert them into a trifocal tensor. This conversion matters. The trifocal tensor does not change under the per-tuple coordinate choice, up to one scale factor, so the network's arbitrary per-tuple frame drops out without any alignment.

Third, we assemble the sparse block tensor. We then compute the leading 6 and 4 singular vectors of its three unfoldings, the matrices formed by flattening the tensor along each mode. For this we use the randomized column sampling that QuadSync showed works with as few as 30 columns. Cameras are read off the factors, and the unknown per-block scales are solved in closed form as in the HOSVD-HT construction. Fourth, we compute the certificate

$$r = \frac{\lVert P_{\Omega}(T - \hat T) \rVert_F}{\lVert P_{\Omega}(T) \rVert_F},$$

where $T$ is the assembled tensor, $\hat T$ its rank-$(6,4,4)$ projection, $P_{\Omega}$ keeps only the observed triplet blocks, and $\lVert\cdot\rVert_F$ is the sum-of-squares norm. We also compute per-view residuals by restricting $\Omega$ to blocks that involve a given view. If $r$ falls below a threshold $\tau$, we output the result directly. Otherwise we run a fixed small number of global-alignment or bundle-adjustment iterations, warm-started from the projected cameras and restricted to the flagged views. We set $\tau$ on held-out scenes by fixing a target failure rate.

We will also analyze two things. The first is how the residual behaves when a single triplet prediction is corrupted, which tells us whether per-view residuals localize errors. The second is where the one-shot projection breaks down. We expect it to fail when triplet coverage is too sparse for the unfoldings to reach full rank, a limitation that both tensor-synchronization papers acknowledge. Rotations come from the network predictions, optionally cleaned by rotation averaging. This choice responds to the finding in Tensor-Based Synchronization that its location estimates are sensitive to sign ambiguities and to the quality of the prior rotations.

Baselines

The naive reference point is DUSt3R with its full global alignment. This is what our method replaces. We run it with its default schedule on the same image sets and GPU and report wall-clock time and pose error. It defines the accuracy we must match and the latency we must beat. The file gives no DUSt3R number on ETH3D ("no reported result found on Lacuna" for that pairing).

The strongest feed-forward competitor is MV-DUSt3R+ run on all views at once. The paper reports reconstructing 12 views in about 0.89 s and 20 views in about 1.54 s, a 48x–78x speedup over DUSt3R with global alignment, and a Chamfer distance about 2.6x lower than its baseline on large HM3D scenes. We run the same checkpoint that produces our triplet predictions. Any gain is then attributable to the tensor projection and gating, not to a better network. Beating it means lower pose error on ETH3D and on the near-collinear sequences at comparable latency, or equal accuracy with a certificate whose flagged scenes account for most of its large errors.

Among iterative geometric methods, we compare against three published systems:

  • QuadSync, reported state-of-the-art or near it in location accuracy on 7 of 11 ETH3D and 4 of 6 EPFL datasets.
  • Cycle-Sync, reported median location error below 0.05 on ETH3D.
  • VGGSfM, reported 33.96% completeness at 5 cm on ETH3D and about 5.4 s for 25 frames.

These set the accuracy ceiling. We run the released code under their published settings with matched hardware. We do not expect to beat their accuracy everywhere. The target is to fall within their error on most scenes at a fraction of their time.

Two ablations isolate our ingredients. The first replaces our one-shot projection with iterative HOSVD-HT on the same neural tensor, which measures what dropping iterations costs. The second synchronizes pairwise DUSt3R poses by linear least squares, which measures whether trifocal structure adds anything beyond pairs. For the gate, we compare the residual against three triggers under the same refinement budget: DUSt3R's per-pixel confidence averaged per scene, the pairwise reprojection residual, and random selection.

Why not just…?

Why not just synchronize DUSt3R's pairwise poses and skip triplets? Pairwise directions carry no consistent scale. QuadSync argues they suffer a rank drop under straight-line motion, and HOSVD-HT beat pairwise methods such as LUD on 8 of 14 Photo Tourism scenes (Tensor-Based Synchronization). There is a counterargument. Optimal motion estimation from multiple images by normalized epipolar constraint holds that properly normalized pairwise constraints make higher-order ones redundant, though its Newton solver spends over 95% of its time on Hessians. Our pairwise ablation settles this empirically. If it matches the trifocal version on the ETH3D near-collinear sequences, the trifocal claim fails and we would say so.

Why not just trust MV-DUSt3R+ end to end? It may well be accurate on average. Our contribution does not require beating it on average. It requires detecting the scenes where it is wrong, particularly those with weak overlap with its reference views, a failure its authors acknowledge. If the residual flags errors no better than the network's own confidence, the certificate adds nothing. The gate comparison tests exactly this.

Why not just always run a few warm-started alignment iterations after the feed-forward pass? This is the most serious threat. A handful of iterations may be cheap enough that gating saves little. We will plot accuracy against average latency for three policies: always running $K$ iterations, gating, and never refining. We report the regime where gating dominates, if one exists. If always-refine with small $K$ matches gating at equal average cost, the practical case for the certificate reduces to failure detection alone, and we would frame the result that way.

Proposed Experiments

The main evaluation uses ETH3D, including its SLAM sequences with near-straight-line camera motion; the EPFL scenes FountainP11 and CastleP30; 1DSfM and IMC Photo Tourism for internet photo collections; and CO3Dv2 and ScanNet++ for object and indoor scenes. We add a synthetic straight-line trajectory following Optimal motion estimation, where scale error is known to be fragile. Metrics are median location and rotation error, completeness at 5 cm where applicable, wall-clock time, and peak memory, each measured as the number of views grows.

Three checks decide the proposal. First, on scenes the certificate passes, one-shot projection should be within a small margin of DUSt3R with global alignment in location error while being at least an order of magnitude faster. Second, the residual should rank scenes by true pose error better than confidence or reprojection triggers. We measure this by the area under the curve for detecting scenes above the median error. Third, at matched average latency, gated refinement should beat both MV-DUSt3R+ alone and always-refine with small $K$. A negative outcome on the first check would mean learned triplets are too noisy for a single projection. A negative outcome on the second would mean algebraic consistency adds nothing beyond what networks already know about their own uncertainty. Either result would be worth reporting.