TRACE-GS: On-Policy Trajectory Distillation with Privileged Geometric Conditioning for Sparse-View 3DGS Restoration

Linlian Jiang, Yuchen Xi, Sadman Rakib Pinon, Ruigang Yang, Yang Wang, Xinxin ZuoarXiv 2026 · arXiv

TL;DR

TRACE-GS introduces an on-policy trajectory distillation framework that adapts video diffusion priors for sparse-view 3D Gaussian Splatting (3DGS) restoration. By using a "privileged" teacher trained on dense-view geometry to guide a sparse-view student along its actual inference rollout, it eliminates the compounding errors caused by training-inference mismatch. This results in significantly sharper, artifact-free 3D reconstructions from as few as three views without adding any computational overhead at deployment.

What Problem Does This Paper Address?

3D Gaussian Splatting (3DGS) has revolutionized real-time 3D scene reconstruction, but it struggles heavily when only a few camera views are available (sparse-view settings). Without enough visual constraints, the reconstructed scene ends up plagued by "floaters," severe blur, and broken geometry.

To fix this, recent state-of-the-art methods use pretrained video diffusion models as generative priors to clean up the artifact-laden renderings. However, these methods suffer from a fundamental flaw: exposure bias (or off-policy state mismatch). During training, the diffusion model is supervised using independently forward-noised states derived from clean target sequences. But during inference, the model generates its own trajectory sequentially. Because sparse-view geometry is inherently flawed, the model's denoising predictions are biased from the very first step. These errors compound over the multi-step rollout, leading to accumulated drift and degraded final restorations.

Figure 1: Off-policy mismatch in sparse-view 3DGS restora- tion. Top: supervision at independently forward-noised states misses rollout states, letting deviations accumulate; on- policy supervision covers visited states. Bottom: on-policy restoration recovers sharper detail from 6 views.

Methods

The authors introduce TRACE-GS, a framework built on two core concepts: On-Policy Trajectory Distillation and Learning Using Privileged Information (LUPI).

Figure 2: Overview of TRACE-GS. (A) Given Vs ⊂Vd, we independently reconstruct a 3DGS from each view set and render paired sequences Rs and Rd along the same target trajectory. (B) Built on the same frozen video-diffusion backbone, the teacher and student use a frozen LoRA ϕT and a trainable LoRA ϕS cloned from a shared initialization, and are conditioned on Rd and Rs, respectively, forming a geometry-asymmetric pair. The student rolls out its reverse trajectory, while the teacher is queried at the same student-visited states without its own rollout. (C) At these states, Ltraj aligns velocity predictions, while Lret aligns cross-view retrieval responses. Only ϕS is updated and retained for deployment.
  1. Geometry-Asymmetric Teacher-Student Roles: The framework uses a single video diffusion backbone with two identical-capacity LoRA adapters: a privileged teacher and a deployable student. During training, the teacher is conditioned on renderings from a 3DGS built with dense views (privileged information), while the student is conditioned on renderings from sparse views.
  2. Student On-Policy Rollout: During training, the student simulates actual inference by rolling out its own multi-step denoising trajectory.
  3. Privileged Teacher Supervision: Instead of letting the student drift, the frozen teacher is queried at the exact states the student visits. The teacher provides a clean, reliable denoising target (velocity matching) to correct the student's trajectory.
  4. On-Policy Retrieval Alignment: To ensure the student correctly leverages cross-view context, a novel retrieval loss aligns the student's cross-view attention responses with the teacher's at each visited state.

At deployment, the teacher and dense-view data are completely discarded. Only the optimized student LoRA adapter is retained, acting as a drop-in enhancement stage to generate high-quality pseudo-observations for iterative 3DGS refinement.

Datasets

The model was trained and evaluated using the following datasets:

  • DL3DV: 112 scenes were used to construct the training data. A disjoint set of 28 scenes was used for in-domain evaluation (DL3DV-Benchmark).
  • Mip-NeRF 360: Used for out-of-domain evaluation to test how well the model generalizes to different capture styles and scene scales.
  • NeRFBusters: Used for out-of-domain evaluation, specifically targeting casually captured scenes containing complex thin structures and highly reflective surfaces.

Key Insights and Ideas

  • Fixing the Training Mismatch Matters More Than Complex Architectures: Instead of engineering increasingly complex video diffusion blocks, simply aligning the training distribution with the inference rollout (on-policy distillation) yields massive performance gains.
  • Privileged Information via View Density: While LUPI frameworks often rely on entirely different modalities (like depth maps or semantic labels), TRACE-GS elegantly defines privileged information as simply having more views of the same scene during training.
  • Late-Stage Trajectory Divergence: The authors observed that off-policy and on-policy variants perform similarly in the early steps of denoising. However, they diverge sharply in the later stages. On-policy supervision prevents the model from plateauing, allowing it to resolve fine details like text strokes and sharp object boundaries.

Key Results

TRACE-GS consistently outperforms vanilla 3DGS and existing diffusion-based baselines (Difix3D+, GenFusion, GSFixer) across all datasets and sparsity levels (3, 6, and 9 views).

  • Quantitative Improvements: On the DL3DV-Benchmark with 3 views, TRACE-GS achieves a PSNR of 16.92 dB, outperforming the previous best baseline (GSFixer at 16.21 dB) and vanilla 3DGS (13.72 dB). The performance gap widens as the number of views decreases, proving the robustness of the privileged guidance under severe sparsity.
  • Qualitative Improvements: TRACE-GS successfully preserves fine structures, sharp text, and complex reflections where other models introduce blur or fragment the geometry.

Qualitative comparison on DL3DV Qualitative comparison on out-of-domain datasets

Limitations and Future Work

  • Static Scenes Only: The current formulation is restricted to static scenes, though the authors note that the framework can naturally extend to dynamic captures.
  • Uncertainty-Aware Supervision: Currently, the teacher's guidance is treated as absolute truth. However, even dense-view reconstructions can have errors in highly challenging regions. Incorporating uncertainty estimation into the teacher's targets remains an open direction for future work.

Summary updated

Abstract

We present TRACE-GS, an on-policy trajectory distillation framework that leverages privileged geometric conditioning at training time, thereby adapting a diffusion prior to sparse-view 3D Gaussian Splatting (3DGS) restoration. Rather than pursuing increasingly sophisticated restoration architectures, we identify a more fundamental limitation shared by existing diffusion-based approaches: supervision at independently noised states does not cover those reached during inference. In sparse-view 3DGS, under-constrained geometry biases denoising from the outset, and the resulting deviations compound along the rollout. TRACE-GS instead performs on-policy trajectory distillation: a teacher conditioned on richer geometry from additional training views supplies targets along the sparse-view student's own rollout, aligning denoising directions and cross-view responses at each visited state. This training-only geometry places TRACE-GS in the learning using privileged information (LUPI) setting. At deployment, only the sparse-view student is retained, and its restored renderings serve as pseudo-observations for 3DGS refinement. To the best of our knowledge, TRACE-GS is the first to derive on-policy supervision from privileged geometry for sparse-view 3DGS restoration, achieving consistent gains and strong generalization across datasets and sparse-view settings.
BibTeX
@misc{jiang2026trace,
  author = {Linlian Jiang and Yuchen Xi and Sadman Rakib Pinon and Ruigang Yang and Yang Wang and Xinxin Zuo},
  title = {{TRACE-GS: On-Policy Trajectory Distillation with Privileged Geometric Conditioning for Sparse-View 3DGS Restoration}},
  year = {2026},
  eprint = {2608.10286},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.10286},
}
Figure