CueSplat: Learning to Schedule Sparse Deformation for Dynamic Gaussian Splatting on Edge Devices

Qingyuan Tang, He Zhu, Shixuan Sun, Anbang Wu, Jingwen Leng, Minyi Guo, Yu FengarXiv 2026 · arXiv

TL;DR

CueSplat is a post-training acceleration framework that brings high-quality dynamic Gaussian Splatting to edge devices by selectively evaluating deformation networks. By predicting Gaussian trajectories from recent history and using a lightweight "router" to prioritize exact updates for only the most critical Gaussians, it achieves up to 8.15× speedups on NVIDIA Jetson hardware with negligible loss in visual quality.

What Problem Does This Paper Address?

Dynamic scene reconstruction using 4D Gaussian Splatting (4DGS) has revolutionized how we capture and render moving scenes. However, these models rely on "deformation networks" that calculate the position, rotation, and scale of millions of Gaussians for every single frame.

On edge devices like the NVIDIA Jetson AGX Orin, this deformation step is a massive bottleneck, often consuming up to 85% of the total GPU pipeline time. While previous methods tried to solve this by pruning Gaussians or pre-calculating static vs. dynamic parts of a scene, these approaches often compromise reconstruction quality or lack the flexibility to adapt to complex motions.

Methods

CueSplat introduces a Predictive Sparse Deformation framework. Instead of asking the heavy deformation network to update every Gaussian at every frame, CueSplat uses a "predict-then-correct" strategy:

  1. Full Prediction: For every frame, the system predicts the state of all Gaussians using a damped finite-difference extrapolation based on the three most recent frames.
  2. Learned Selection (The Router): A lightweight MLP router identifies which Gaussians are likely to have high prediction errors or contribute most significantly to the final image (based on visibility and opacity).
  3. Selective Update: Only a small, scene-specific "budget" of Gaussians (e.g., the top 20%) receives an exact update from the frozen pretrained deformation network. The rest use their predicted states.
  4. Index Reuse: To save on computation, the router only re-evaluates which Gaussians to update every $N=5$ frames, reusing those indices in between.
  5. Adaptive Refresh: To prevent "drift" (accumulated prediction error), CueSplat monitors residuals and periodically triggers three consecutive full-deformation frames to rebuild a clean history for the predictor.
Overview of the CueSplat pipeline, showing offline preparation and the online inference loop with selective deformation and adaptive refresh.

Datasets

The authors evaluated CueSplat across three widely used dynamic scene benchmarks:

  • N3DV: 9 scenes featuring complex human motions and interactions.
  • Technicolor: 5 scenes captured in a multi-camera studio environment.
  • NeRF-DS: 7 sequences focusing on dynamic specular objects.

The framework was tested against four mainstream 4DGS backbones: E-D3DGS, 4DGaussians, GaGS, and SpeeDe3DGS.

Key Insights and Ideas

  • Local Temporal Regularity: The authors observed that while global motion is complex, individual Gaussian trajectories are often locally predictable (linear or quadratic) over short windows. As shown in Figure 2, over 66% of Gaussian windows in N3DV follow simple regular motion.
  • Reliability vs. Contribution: Not all prediction errors are equal. An error in a Gaussian that is occluded or has low opacity doesn't hurt the final render. The router successfully learns to balance "prediction risk" with "rendering importance."
  • Post-Training Compatibility: One of CueSplat's biggest strengths is that it is a post-training framework. It works with existing, frozen 4DGS models, meaning researchers don't need to redesign or retrain their complex deformation backbones from scratch.
Visualization of projected Gaussian trajectories. Green marks show locally linear paths, while red marks show nonlinear turns where prediction is harder.

Key Results

CueSplat demonstrates that you can trade a tiny amount of PSNR for a massive gain in throughput:

  • Throughput Gains: It achieved average speedups of 1.75× to 6.74× on Jetson AGX Orin and up to 8.15× on Orin NX.
  • Visual Fidelity: The drop in quality was nearly imperceptible, with a maximum dataset-average PSNR decrease of only 0.16 dB.
  • Computational Efficiency: Server-side profiling showed a 29.9% to 90.7% reduction in executed GPU warp instructions per frame.
Comparison of FPS across different backbones. CueSplat (dark green) significantly outperforms the baselines (light green) across all tested models.

Limitations and Future Work

  • Sequential Dependency: Because the system relies on a three-frame history for prediction, it is optimized for sequential playback. "Temporal seeking" (jumping to a random timestamp) requires a brief re-initialization of the history.
  • Scene-Specific Calibration: The framework requires a short offline calibration phase (1–2 minutes) to determine the optimal update budget and train the router for a specific scene.
  • Future Directions: The authors suggest exploring variable playback rates and further reducing the costs associated with scene-specific preparation to make the system even more "plug-and-play."

Summary updated

Abstract

Deformation-based 4D Gaussian splatting (4DGS) enables dynamic scene rendering. However, evaluating deformation networks for all Gaussians at every frame is compute-intensive on edge devices. We find that Gaussian trajectories are often locally predictable, yet prediction reliability varies across Gaussians and time. Thus, we propose CueSplat, a post-training framework that exploits local temporal regularity while keeping the pretrained backbone frozen and retaining its Gaussian set. Specifically, CueSplat predicts Gaussian attributes from recent history and uses a lightweight learned router to select exact deformation updates within a scene-specific budget calibrated offline. Selection accounts for prediction reliability and rendering contribution. Selected indices are reused across nearby frames to reduce routing overhead, while selected Gaussians still receive exact updates each frame. To control accumulated prediction error, we use drift signals to schedule three consecutive full-deformation frames that rebuild the prediction history. Evaluated across four mainstream 4DGS algorithms, CueSplat achieves up to $6.74\times$ and $8.15\times$ speedups on Jetson AGX Orin and Orin NX, respectively, with a negligible quality drop (at most $0.16$ dB in dataset-average PSNR).
BibTeX
@misc{tang2026cuesplat,
  author = {Qingyuan Tang and He Zhu and Shixuan Sun and Anbang Wu and Jingwen Leng and Minyi Guo and Yu Feng},
  title = {{CueSplat: Learning to Schedule Sparse Deformation for Dynamic Gaussian Splatting on Edge Devices}},
  year = {2026},
  eprint = {2611.22095},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22095},
}
Figure