TL;DR
TRACE-GS introduces an on-policy trajectory distillation framework that adapts video diffusion priors for sparse-view 3D Gaussian Splatting (3DGS) restoration. By using a "privileged" teacher trained on dense-view geometry to guide a sparse-view student along its actual inference rollout, it eliminates the compounding errors caused by training-inference mismatch. This results in significantly sharper, artifact-free 3D reconstructions from as few as three views without adding any computational overhead at deployment.
What Problem Does This Paper Address?
3D Gaussian Splatting (3DGS) has revolutionized real-time 3D scene reconstruction, but it struggles heavily when only a few camera views are available (sparse-view settings). Without enough visual constraints, the reconstructed scene ends up plagued by "floaters," severe blur, and broken geometry.
To fix this, recent state-of-the-art methods use pretrained video diffusion models as generative priors to clean up the artifact-laden renderings. However, these methods suffer from a fundamental flaw: exposure bias (or off-policy state mismatch). During training, the diffusion model is supervised using independently forward-noised states derived from clean target sequences. But during inference, the model generates its own trajectory sequentially. Because sparse-view geometry is inherently flawed, the model's denoising predictions are biased from the very first step. These errors compound over the multi-step rollout, leading to accumulated drift and degraded final restorations.
Methods
The authors introduce TRACE-GS, a framework built on two core concepts: On-Policy Trajectory Distillation and Learning Using Privileged Information (LUPI).
- Geometry-Asymmetric Teacher-Student Roles: The framework uses a single video diffusion backbone with two identical-capacity LoRA adapters: a privileged teacher and a deployable student. During training, the teacher is conditioned on renderings from a 3DGS built with dense views (privileged information), while the student is conditioned on renderings from sparse views.
- Student On-Policy Rollout: During training, the student simulates actual inference by rolling out its own multi-step denoising trajectory.
- Privileged Teacher Supervision: Instead of letting the student drift, the frozen teacher is queried at the exact states the student visits. The teacher provides a clean, reliable denoising target (velocity matching) to correct the student's trajectory.
- On-Policy Retrieval Alignment: To ensure the student correctly leverages cross-view context, a novel retrieval loss aligns the student's cross-view attention responses with the teacher's at each visited state.
At deployment, the teacher and dense-view data are completely discarded. Only the optimized student LoRA adapter is retained, acting as a drop-in enhancement stage to generate high-quality pseudo-observations for iterative 3DGS refinement.
Datasets
The model was trained and evaluated using the following datasets:
- DL3DV: 112 scenes were used to construct the training data. A disjoint set of 28 scenes was used for in-domain evaluation (DL3DV-Benchmark).
- Mip-NeRF 360: Used for out-of-domain evaluation to test how well the model generalizes to different capture styles and scene scales.
- NeRFBusters: Used for out-of-domain evaluation, specifically targeting casually captured scenes containing complex thin structures and highly reflective surfaces.
Key Insights and Ideas
- Fixing the Training Mismatch Matters More Than Complex Architectures: Instead of engineering increasingly complex video diffusion blocks, simply aligning the training distribution with the inference rollout (on-policy distillation) yields massive performance gains.
- Privileged Information via View Density: While LUPI frameworks often rely on entirely different modalities (like depth maps or semantic labels), TRACE-GS elegantly defines privileged information as simply having more views of the same scene during training.
- Late-Stage Trajectory Divergence: The authors observed that off-policy and on-policy variants perform similarly in the early steps of denoising. However, they diverge sharply in the later stages. On-policy supervision prevents the model from plateauing, allowing it to resolve fine details like text strokes and sharp object boundaries.
Key Results
TRACE-GS consistently outperforms vanilla 3DGS and existing diffusion-based baselines (Difix3D+, GenFusion, GSFixer) across all datasets and sparsity levels (3, 6, and 9 views).
- Quantitative Improvements: On the DL3DV-Benchmark with 3 views, TRACE-GS achieves a PSNR of 16.92 dB, outperforming the previous best baseline (GSFixer at 16.21 dB) and vanilla 3DGS (13.72 dB). The performance gap widens as the number of views decreases, proving the robustness of the privileged guidance under severe sparsity.
- Qualitative Improvements: TRACE-GS successfully preserves fine structures, sharp text, and complex reflections where other models introduce blur or fragment the geometry.

Limitations and Future Work
- Static Scenes Only: The current formulation is restricted to static scenes, though the authors note that the framework can naturally extend to dynamic captures.
- Uncertainty-Aware Supervision: Currently, the teacher's guidance is treated as absolute truth. However, even dense-view reconstructions can have errors in highly challenging regions. Incorporating uncertainty estimation into the teacher's targets remains an open direction for future work.
Summary updated