3D Gaussian Splatting represents a scene as millions of small, colored, semi-transparent ellipsoids that can be drawn to the screen very fast. Dynamic versions move these ellipsoids over time to capture people and objects in motion. How fast the scene renders depends mostly on how many ellipsoids there are. How many there are depends on densification, the training heuristic that splits and clones ellipsoids wherever the rendered image disagrees strongly with the photo.
Our observation is that densification cannot tell why the image disagrees. A camera pose that is slightly wrong shifts every edge in the frame. To densification this looks like unresolved fine detail everywhere, so it spawns ellipsoids to paper over the misalignment. We therefore propose to treat camera pose error as a direct cause of rendering latency in online dynamic reconstruction. The method has three parts. First, it re-estimates each incoming frame's pose using only the static part of the scene. Second, it lets the dynamic motion model and a cheap 2D noise map explain what remains. Third, it allows densification to act only on the residual that survives both. The claim to test is that this keeps the ellipsoid count, and therefore frame rate and memory traffic, close to what one would get with perfect poses, while plain online pipelines inflate the count as pose noise grows.
Core Claim
Existing online dynamic splatting assumes calibrated cameras. The closest work, Compensating Spatiotemporally Inconsistent Observations for Online Dynamic 3D Gaussian Splatting (Yun et al.), splits each frame into an ideal rendering plus a learnable, sparsity-penalized 2D residual image. That residual soaks up sensor noise and cuts the Gaussian count by 20–30%, but it does not touch poses. Robust static methods such as RobustSplat delay densification so that moving objects do not become permanent floaters, but they also assume fixed poses. Dynamic SLAM systems do refine poses in moving scenes: DG-SLAM, D2GSLAM, and De4D-SLAM. They measure trajectory error and image quality, not how pose error inflates the model and slows rendering.
Our claim is specific. In online dynamic splatting, residual pose error is converted into extra primitives. Ordering the updates as pose first, then motion, then 2D sink, and only then densification removes most of that conversion. Our prediction is that the 2D sink alone is not enough once misalignment exceeds roughly a pixel. The reason is that a pose shift produces structured, image-wide edge residuals, which a sparsity penalty is designed to reject.
Research Program
The system builds on an online dynamic splatting backbone of the kind Yun et al. wrap, such as 3DGStream. The static background and the moving foreground are held as separate Gaussian sets, following the dual-stream split used in De4D-SLAM and RefiNeRF. For each new frame, the system runs four steps in a fixed order.
The first step is pose alignment. It runs a few iterations of camera-pose optimization over the camera's 6-degree-of-freedom rigid motion, using the fast analytical splatting pose gradients of Schmidt et al.. That paper reports pose refinement about 40× faster than NeRF baselines, which is what makes a per-frame pose step affordable online. The pose step sees only pixels currently attributed to static Gaussians.
The second step updates the dynamic deformation. Following De4D-SLAM's gradient-isolation idea, the dynamic stream is blocked from receiving gradients from the static residual. This stops moving primitives from absorbing leftover pose misalignment.
The third step updates Yun et al.'s 2D residual sink.
The fourth step is densification. It is gated on the view-space positional gradient recomputed after the first three steps, rather than on the raw gradient accumulated during them. Concretely, a Gaussian $i$ is split or cloned only if
$$\big\|\nabla_{\mu_i}\,\mathcal{L}\big(I_t,\ \hat I_t(\hat\xi_t,\hat\theta_t)+\hat M_t\big)\big\| > \tau.$$
Here $\mu_i$ is the Gaussian's projected center and $\mathcal{L}$ is the photometric loss against the captured frame $I_t$. The rendering $\hat I_t$ uses the refined camera pose $\hat\xi_t$ and the updated dynamic deformation $\hat\theta_t$, $\hat M_t$ is the 2D residual image, and $\tau$ is the usual densification threshold. Standard pipelines apply the same test to the gradient before $\hat\xi_t$ and $\hat M_t$ have absorbed their share of the error.
The analysis part of the program is a dose–response study. We inject controlled per-frame rotation and translation errors into otherwise calibrated cameras. We then measure how the Gaussian count, frames per second, and bytes read per frame grow with the error for each pipeline. Memory traffic is the quantity Potamoi identifies as the binding constraint on mobile hardware, where it reports that NeRF and splatting fail to exceed 5 FPS. The study also has a visual-quality side. Zhang et al. and Strecha et al. establish that pose error causes blur and geometric loss, so the dose–response study also tracks visual quality to check whether primitive growth buys any of that quality back.
Baselines
The naive reference is the unmodified online backbone with noisy poses and standard densification. Its twin run with ground-truth poses defines the target count $N_0$ and target frame rate that every other method is measured against.
The strongest existing competitor is Yun et al.'s observation-restoring wrapper on the same backbone. Under calibrated poses it is the only method in the file that directly reduces primitive count in online dynamic splatting: it reports roughly 0.5–1.0 dB PSNR gain, 20–30% fewer Gaussians, and up to 70% less flicker (masked total variation). We found no reported result on Lacuna for it under pose perturbation. We will run it with its published hyperparameters on identical frames, noise draws, backbone, and GPU. Its sparsity weight will be tuned with the same search budget we give our own thresholds.
A second competitor gives the backbone per-frame pose refinement but uses standard densification. We call it "pose-refinement-only." It isolates whether gating densification matters beyond simply fixing poses.
A third competitor is RobustSplat-style delayed densification added to the backbone. It tests whether simply waiting to densify is enough.
The key ablation keeps every one of our components but densifies on the raw, pre-alignment gradient. If it matches the full method, the gating contributes nothing and the claim fails.
Beating the strongest baseline means the following. At matched PSNR, within 0.2 dB, our Gaussian count stays within 10% of $N_0$ at pose-noise levels where Yun et al.'s count exceeds $N_0$ by clearly more than that. Frame rate must rise correspondingly. These are pre-registered targets, not results from any paper.
Why not just…?
Why not just precompute better poses with COLMAP or DROID-SLAM and then run the calibrated pipeline? In online use there is no chance to run global structure-from-motion, and it breaks in the settings we care about. RefiNeRF documents COLMAP failing on low-texture surgical video, and UPGS shows motion blur corrupting initial poses. Even good trackers leave centimeter-level error: DG-SLAM reports 5.5 cm and De4D-SLAM reports 1.95 cm average trajectory error on Bonn. Our Bonn experiment uses exactly such tracker poses, so the question is answered empirically rather than argued.
Why not just let Yun et al.'s 2D residual map absorb the pose jitter? This is the real threat. For sub-pixel jitter it may well suffice, and we will say so if it does. Our hypothesis is that a sparsity-penalized image cannot cheaply represent the image-wide, edge-aligned residual that a pose shift produces, so densification takes over above roughly one pixel of misalignment. The dose–response curve settles this directly. The crossover point, if any, is itself a useful result.
Why not just freeze densification or prune afterward? RobustSplat reports that disabling densification removes floaters but gives overly smooth reconstructions. Post-hoc compaction of the kind surveyed by Ali et al. can restore the count, but the geometry was already fit around misaligned views. We will include "backbone + opacity pruning to $N_0$" as an extra run to check whether it matches our PSNR at equal count.
Proposed Experiments
The controlled study uses Neural 3D Video and MeetRoom. These are multi-camera rigs with trustworthy calibration, onto which we inject per-frame extrinsic noise at several magnitudes. We report PSNR, LPIPS, masked total variation, Gaussian count, FPS, and bytes read per frame.
The real-error study uses Bonn RGB-D Dynamic and TUM RGB-D. Initial poses come from a DROID-style tracker, and we report trajectory error alongside the same efficiency metrics. We will compare pose accuracy against De4D-SLAM's reported 1.95 cm on Bonn only as context, since our goal is primitive economy rather than beating SLAM on trajectory error.
For per-frame latency, we note that the dynamic SLAM systems report 0.47–2 FPS for optimization. We will state our per-frame update time honestly rather than claim real-time mapping.
The decisive check is the Neural 3D Video dose–response plot of Gaussian count against injected pose error for the naive baseline, Yun et al., pose-refinement-only, and the full method. The proposal succeeds if the full method's curve stays flat near $N_0$ while the others rise, and if the raw-gradient ablation rises with them. It fails if Yun et al. or pose-refinement-only already stays flat.
References
- Compensating Spatiotemporally Inconsistent Observations for Online Dynamic 3D Gaussian Splatting
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- DG-SLAM: Robust Dynamic Gaussian Splatting SLAM with Hybrid Pose Optimization
- D2GSLAM: 4D Dynamic Gaussian Splatting SLAM
- De4D-SLAM: Gradient-Isolated Static-Dynamic Decoupling for Monocular SLAM in Dynamic Environments
- RefiNeRF: Modelling dynamic neural radiance fields with inconsistent or missing camera parameters
- Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization
- Potamoi: Accelerating Neural Rendering via a Unified Streaming Architecture
- Digging into Radiance Grid for Real-Time View Synthesis with Detail Preservation
- On benchmarking camera calibration and multi-view stereo for high resolution imagery
- UPGS: Unified Pose-aware Gaussian Splatting for Dynamic Scene Deblurring
- Compression in 3D Gaussian Splatting: A Survey of Methods, Trends, and Future Directions