Transport Consensus: Fusing Disagreeing Generated Views into One 3D Object

Fast text-to-3D systems now work in two steps. A multi-view diffusion model, a generator that learns to turn random noise into images by gradually denoising them, produces a few views of an object. A feed-forward reconstructor then lifts those views into 3D Gaussians, which are small colored ellipsoids that together render the scene, in a fraction of a second. The weak link is the handoff. Generated views are never exactly consistent: an ear sits a few pixels higher in one view than in its neighbour, or a limb is slightly rotated. The reconstructor was trained on consistent renderings, so when two views disagree it tends to keep both versions at partial opacity. The result is ghosted doubles, floaters, and duplicated parts. These are the reconstruction-stage cousins of the Janus problem, the tendency of 2D-guided 3D generation to sprout extra faces and limbs.

We propose to fix this handoff with a small, training-free layer whose one guiding idea is easy to state. Averaging two disagreeing shapes the way pixels are averaged gives two half-transparent copies. Averaging them the way optimal transport does, by moving mass from one to the other, gives one solid copy halfway between. Concretely, the layer groups the predicted Gaussians by the input view they came from. It matches them across views with unbalanced optimal transport and replaces each matched group with its Wasserstein barycenter, the single Gaussian that is cheapest to "move" all the others into. Unmatched Gaussians are kept, or have their opacity reduced, rather than being forcibly averaged. The leftover transport cost after matching gives, at no extra cost, a map of where the generated views disagree. We test whether this beats both the raw pipeline and the expensive fixes the field currently uses: reinforcement-learning fine-tuning of the diffusion model, and hours of per-scene optimization.

Core Claim

Cross-view inconsistency in generated multi-view images turns into duplicated or ghosted 3D structure because current reconstructors fuse overlapping evidence with what amounts to an additive, mixture-style average over Gaussian densities. Fusing it instead with a transport-based average, computed directly on the Gaussian parameters, removes much of this structure at feed-forward cost. It also produces a disagreement signal that can stand in for reconstruction-and-re-render consistency scores. The claim is deliberately narrow. It targets artifacts that come from views disagreeing about the same region of space. It does not target a "global" Janus object whose views consistently agree that it has two faces, because no fusion rule can repair what all views assert. We will measure both kinds separately so the claim cannot be overstated.

Why this, and why now

Four lines of work set up the opportunity. MVDream (MVDream) adds cross-view attention and camera embeddings to a diffusion model and largely removes the multi-face failure. However, it still lifts to 3D through roughly two hours of score distillation per asset on a V100, versus about 1.5 hours on four TPUv4 chips for the original DreamFusion. Its residual inconsistency is what our layer is meant to absorb. The feed-forward reconstructors supply the fast lifting step we build on. GS-LRM maps 2–4 posed images to per-pixel Gaussians in 0.23 seconds, and ReSplat refines them over four recurrent render-and-correct steps. Both assume their input views are mutually consistent, as real captures are. That assumption is exactly what generated inputs break.

Carve3D motivates the proposal most directly and is also its main competitor. It scores a multi-view generation by reconstructing it with a fast reconstructor, re-rendering, and measuring LPIPS against the inputs (the Multi-view Reconstruction Consistency, or MRC, metric). It then RL-fine-tunes a 2.6B-parameter diffusion model on that score, cutting MRC on the 415-prompt DreamFusion set from 0.0892 to 0.0606. Its authors flag the RL loop's cost as unresolved. Carve3D shows the inconsistency is real and measurable. It repairs the generator, while we repair the fusion and leave the generator untouched. The two are complementary, but they also compete for the same benefit, so we compare them head-on.

The Wasserstein side supplies the method. Splat Regression Models shows that fitting Gaussian splats is a discrete Wasserstein–Fisher–Rao gradient flow. Here "Wasserstein" governs moving and reshaping primitives, and "Fisher–Rao" governs creating and destroying their mass. That split is exactly the matched-versus-unmatched decision our layer makes. Gradient descent algorithms for Bures-Wasserstein barycenters proves that the barycenter objective on non-degenerate Gaussians satisfies a Polyak–Łojasiewicz-type inequality. This gives gradient descent a linear rate $(1-\zeta^2/4)^T$, where $\zeta$ shrinks as covariances become degenerate. The result means a handful of fixed-point iterations per matched group suffices. It also warns us that flat, needle-like splats will converge slowly, so we must regularize them.

Two papers already put $W_2$ to work on Gaussian splats, and both constrain our design. WaSt-3D matches style and content Gaussians with Sinkhorn divergence for stylization. Its 8-minute, 16 GB per-scene optimization and its reliance on K-means clustering show that global optimal transport over millions of splats is too slow for our setting. Gaussians on their Way uses $W_2$ to regularize Gaussian motion over time in 4D reconstruction and reports up to 57% shorter training. Neither touches generated inputs or cross-view consensus, and both remain per-scene optimizers.

Research Program

The pipeline is a frozen multi-view diffusion model, then a frozen GS-LRM-style reconstructor, then the consensus layer. The reconstructor's per-pixel outputs come pre-labelled by source view $k$. Each view therefore defines a weighted set of Gaussians $\{(w_i, \mu_i, \Sigma_i)\}$, with opacity weight $w_i$, mean $\mu_i$, and covariance $\Sigma_i$. The cost between two Gaussians combines the closed-form Bures–Wasserstein distance with a color term:

$$ c(i,j) = \|\mu_i-\mu_j\|^2 + \mathrm{tr}\big(\Sigma_i+\Sigma_j-2(\Sigma_i^{1/2}\Sigma_j\Sigma_i^{1/2})^{1/2}\big) + \lambda\|r_i-r_j\|^2, $$

where $r$ is the Gaussian's color and $\lambda$ weights color against geometry.

For each pair of views with overlapping frusta, we solve an entropic, unbalanced transport problem between the two Gaussian sets. "Unbalanced" means mass that has no good partner in the other view is left unmatched at a price $\tau$, rather than being forced into a match. To keep this cheap, transport is restricted to local cells along each ray's depth window instead of over the whole scene. This replaces WaSt-3D's global clustering with geometry the camera already gives us. Matched groups are replaced by their Bures–Wasserstein barycenter. The covariance is computed by the fixed-point iteration whose convergence the barycenter paper establishes, warm-started from the Euclidean average. Opacity is taken from the matched mass. Unmatched mass in regions seen by only one view is kept. Unmatched mass in regions seen by several views, where the other views contradict it, has its opacity reduced in proportion to the contradiction; this is the Fisher–Rao part.

The per-pixel residual transport cost, rendered into each view, is our disagreement map. Its scene-level mean is a scalar consistency score, which we call transport residual. We use it in one place only: choosing the best of $N$ diffusion seeds, so that we can compare it fairly against using MRC for the same purpose. A second, optional stage trains a light reconstructor head with the layer in the loop on inconsistency-augmented Objaverse data. Its purpose is to test whether the gains survive once the reconstructor has learned to cope with inconsistent views on its own.

Baselines

The naive reference point is the same frozen diffusion model feeding the same frozen reconstructor with no consensus step, essentially the Instant3D-style pipeline Carve3D starts from. Every comparison uses identical prompts, seeds, view counts, and reconstructor weights. Beating it means lower MRC, fewer human-flagged ghosted or duplicated parts, and no loss on T3Bench alignment, at under one second of added latency.

The strongest existing method is Carve3D's RL-fine-tuned generator, which reports MRC 0.0606 against 0.0892 for its base on the DreamFusion 415-prompt set (Carve3D). If its weights and base model are available, we evaluate both pipelines with the same reconstructor used for MRC scoring and compare directly. We also stack our layer on top of Carve3D. If the weights are not available, we reproduce its recipe (30 prompts, KL-regularized policy gradient) on our open multi-view base under a matched GPU-hour budget. The published numbers are then treated only as a reference, since they come from a different base model. Beating it means matching its MRC without any generator training, or improving on it when stacked.

The second strong competitor is short test-time refinement. We consider ReSplat-style recurrent correction (ReSplat) and a few hundred steps of standard photometric 3D Gaussian Splatting optimization against the generated views, each given the same wall-clock budget as our layer and also ten times that budget. For per-scene distillation we cite MVDream's roughly two hours per asset on a V100 as the cost ceiling. No reported result was found on Lacuna for MVDream's MRC or for T3Bench numerical scores, so we will measure those ourselves.

Three ablations isolate the key ingredient. The first uses the same transport matching but Euclidean averaging of means and covariances, testing whether the barycenter matters or only the matching. The second skips matching entirely and averages opacity across overlapping splats, which is the mixture behaviour we claim causes ghosting. The third uses balanced transport, which removes the Fisher–Rao mass decision.

Why not just…?

Why not just fine-tune the reconstructor on deliberately inconsistent inputs? This is the most serious threat. A transformer that attends across views could learn to resolve disagreements itself. We will train exactly that baseline, using the same augmentations (per-view part displacements, small warps, color shifts) as our optional second stage. We report results on both synthetic and generated inconsistency. If the augmented reconstructor matches the consensus layer, the honest conclusion is that augmentation suffices. In that case the remaining contribution reduces to the disagreement map, and we will say so. Our expectation is that learned fusion still blends under distribution shift, but the experiment decides.

Why not just use Carve3D's RL fine-tuning? It works, as its MRC drop shows, but it requires a training loop over a 2.6B-parameter model that its authors call expensive. It must also be redone for every new generator. The head-to-head and stacked comparisons above settle whether a free fusion step captures most of the benefit. If RL clearly dominates even after stacking, the layer's case rests on cost alone.

Why not just run a short per-scene optimization or recurrent refinement? Photometric fitting to inconsistent targets is a least-squares compromise, which is the same averaging we argue produces ghosts. Wasserstein-regularized fitting in Gaussians on their Way helped precisely where Euclidean penalties caused tearing and flicker. Still, refinement with more compute might win on sharpness. The matched-budget and ten-times-budget runs show where the crossover lies.

Why not just use view-dependent prompts? DreamFusion itself reports that viewpoint prompting does not reliably remove multi-face artifacts. In any case it acts on the generator, so it is orthogonal and can be applied to every arm.

Proposed Experiments

The first experiment is controlled. We render GSO and Objaverse objects from four views, inject known per-view inconsistencies of increasing magnitude, and measure held-out-view PSNR and LPIPS against the clean ground truth. We also count duplicated parts automatically, by checking whether the reconstruction has two separated components where the ground truth has one. The decisive check is whether quality as a function of inconsistency magnitude degrades more slowly with transport consensus than with the naive and Euclidean-merge arms. It should, because mixture averaging produces two modes once displacement exceeds splat size, while the barycenter produces one.

The second experiment is generative. We run the full pipeline on the DreamFusion 415-prompt set and the 300-prompt T3Bench. We report MRC, T3Bench quality and alignment scores (whose ratings correlate with human experts above 0.75), wall-clock time per asset, and a blinded human study that separately labels local ghosting or duplication and global multi-face Janus. The claim predicts gains on the first and little change on the second, and we will report it as falsified if local artifacts do not drop. We break results down across T3Bench's single-object, with-surroundings, and multi-object splits, since the benchmark found the harder splits degrade sharply and overlapping-object regions are where cross-view disagreement should be worst.

The third experiment tests the transport residual as a seed-selection score. With a fixed budget of $N$ seeds, we compare picking by transport residual against picking by MRC and against random choice. We measure each pick's resulting MRC and human preference and report the scoring time. A residual that selects as well as MRC without a re-render pass gives a cheaper selection signal and a cheaper candidate RL reward, though we leave the RL use to future work. Finally, we log failures where needle-like covariances make barycenter iterations slow. This tests whether the degeneracy warning from the Bures–Wasserstein convergence theory shows up in practice, and whether covariance flooring removes it.