Spending Bits or Compute: Joint Scheduling of Gaussian-Splat Streaming and On-Device Restoration

Streaming a 3D scene to a VR headset now means sending millions of small colored blobs (3D Gaussian splats) over a wireless link that cannot carry them all in time. Current systems pick which blobs to send first based on what the viewer is about to look at. When bandwidth runs out, they simply show the coarser scene. Meanwhile, work on efficient image restoration has produced networks and lookup tables that sharpen blurry images at a fraction of the usual compute. These two lines of work have never met. This proposal treats the headset's spare compute as a second budget alongside network bandwidth. For every region of the scene, it decides whether to spend bits (download the missing detail) or milliseconds (restore that region on the device from what already arrived).

The central observation is that restoration substitutes for bits well in some places and badly in others. A wall rendered from coarse splats is blurry in a way a small network can fix. A thin railing whose fine splats were never sent has missing geometry that no 2D network can reconstruct. If the server measures, offline, how much a lightweight restorer recovers for each object, the client can solve one scheduling problem over both bandwidth and compute. It would download detail where restoration fails and restore where it succeeds. In two sentences: stream fewer splats and sharpen the rendered frame on-device, but choose per region which of the two to use, using a "restorability" score computed ahead of time from the scene itself. The restorer is further told, per pixel, which pixels came only from coarse splats, so it spends its effort only there.

Core Claim

At matched network bitrate and matched on-device restoration time, a scheduler that jointly allocates bits and restoration compute per object will give higher fidelity along real 6-degree-of-freedom (6DoF) head trajectories than either of two alternatives. The first is a bandwidth-only layered scheduler. The second is the same scheduler followed by an independent restoration pass. The gain should come specifically from the coupling. The decoupled pipeline is therefore the baseline that decides whether this idea is worth anything.

Where This Sits

The streaming side supplies the machinery. L3GS splits a Gaussian scene into a base layer and enhancement layers per semantic object, predicts the viewport by linear regression, and solves a knapsack over bandwidth every second on a Meta Quest 3. It reports 16.9% higher average SSIM than importance sorting, with under 3 ms of scheduling overhead. We reuse its layering and knapsack and add a second resource and a second action per object. Adaptive 3D Gaussian Splatting Video Streaming chooses quality tiers per tile with an integer program that accounts for bandwidth and frame rate but not on-device compute. PRoGS orders splats by contribution for progressive loading and reaches 21.33 dB on Bonsai at 10% of the data. All three leave the client as a plain rasterizer. That gap motivates this work.

The restoration side supplies the compute-saving techniques and also constrains the design. Restore Globally, Refine Locally (MGA) sends only the hardest patches, predicted from 2D features, to a deeper refinement network. It cuts FLOPs by 10–48% and runs 22% faster on a Kirin 980. Its masks know nothing about the 3D cause of blur, which is the ingredient we replace. NAFNet shows that a plain UNet with no nonlinear activations, using channel-splitting products and simplified channel attention, matches heavier restorers. We build the restorer from these blocks. MuLUT replaces arithmetic with factorized lookup tables at about 278 μJ per HD image, but it takes roughly 240 ms per HD image on a Snapdragon 888 CPU. That rules out full-frame lookup-table restoration at headset frame rates and forces region-limited restoration.

The closest connecting work is SplatSuRe. It scores each Gaussian by how large it appeared in training views and applies 2D super-resolution supervision only to the 10–20% of pixels that were undersampled. This improves Tanks & Temples 4× results from 23.32 dB (SRGS) to 23.81 dB. SplatSuRe shows that geometry-derived selectivity beats uniform super-resolution, which supports our per-pixel conditioning. However, it runs offline during scene fitting and produces a fixed model. It has no notion of bandwidth, viewer motion, or client compute. SRGS roughly doubles the number of primitives, the opposite of what a bandwidth-limited stream needs. Cross-Resolution Flow Propagation restores foveated 2D video and caches features from past gaze regions, but it aligns frames with 2D deformable convolutions. We borrow its feedback idea and replace 2D alignment with reprojection through rendered depth.

Research Program

The work has three parts.

The first is an offline restorability profile computed on the server. For each object and each enhancement level, we render training views three ways: base-only, base plus restoration, and full quality. The result is a per-object table of fidelity gained per downloaded byte versus fidelity gained per millisecond of restoration, conditioned on viewing distance bins. The profile is small metadata sent with the base layer. If restorability turns out to be predictable from cheap splat statistics, such as the ratio of screen-space footprint to missing-enhancement splat count or opacity residuals, the table can be replaced by a regressor.

The second is a geometry-conditioned restorer. It is a NAFNet-style network of a few blocks. Its input is the rendered frame plus three auxiliary buffers the rasterizer produces for free: depth, per-pixel splat footprint, and a map marking pixels covered only by base-layer splats. It runs only on the pixels the scheduler assigned to restoration. To avoid the patch slicing and re-stitching overhead that erodes MGA's gains on mobile GPUs, regions are processed as fixed-size tiles aligned to the object masks. The restorer also receives the previous restored frame, reprojected through rendered depth and the known camera motion. This targets the flicker that per-frame 2D restoration causes and replaces the 2D flow that breaks under parallax.

The third is a two-budget scheduler. The L3GS knapsack is extended so that each object in the predicted viewport has three options: download the next enhancement layer, restore, or leave it. Bandwidth, predicted by harmonic mean, and restoration time per frame, measured on-device, are separate constraints. Because both costs are per object and the restoration budget resets each frame, this remains a small multi-constraint knapsack solvable greedily within the existing millisecond overhead. The same structure also handles thermal throttling: when the device slows, the compute budget shrinks and the scheduler shifts work back toward bits.

Baselines

The naive reference renders whatever layers a bandwidth-only greedy scheduler delivered, with no restoration. It sets the floor for every trace and bitrate.

The strongest existing streaming method is L3GS. We run its layering, viewport predictor, and knapsack on the same scenes, the same Quest 3 6DoF traces, and the same bandwidth traces. The only thing we tune is its layer count, using the same budget of configurations as our method. Its reported figure is 16.9% higher average SSIM than importance sorting. We will reproduce its absolute SSIM in our setup rather than borrow a number. No absolute per-scene L3GS SSIM is reported in the file.

The strongest decoupled combination is L3GS scheduling followed by restoration of the whole frame, run once with a small NAFNet and once with MuLUT, at the same per-frame millisecond budget as our method. A 2D-mask variant replaces full-frame restoration with an MGA-style mask predictor. For MGA the file reports 22% speedup on a Kirin 980 and 10–48% FLOP reduction. No reported result was found on Lacuna for MGA, NAFNet, or MuLUT running inside a splat-streaming client.

The ablations isolate the key ingredients. The first keeps our restorer and buffers but uses the decoupled scheduler, which isolates joint allocation. The second keeps joint allocation but removes the geometry buffers, which isolates 3D conditioning. The third removes depth reprojection, which isolates temporal stability.

Beating the baselines means Pareto dominance on the bits-versus-fidelity curve at a fixed per-frame restoration budget. We would count as meaningful either the same trajectory-averaged SSIM and LPIPS as L3GS at 20% or more fewer bits, or a statistically reliable gain over the decoupled L3GS-plus-restorer pipeline at equal bits and equal milliseconds, with no increase in measured flicker.

Why not just…?

Why not just drop splats and use level-of-detail, with no restoration at all? This is the naive baseline, and it costs zero client compute. The file's evidence that it leaves quality on the table is suggestive, not decisive. PRoGS stays at 21.33 dB with 10% of the data, and SRGS shows that 3DGS fitted from low-resolution views reaches only 20.31 dB against 30.83 dB with restoration-derived supervision. Neither result measures real-time restoration of a streamed base layer. The first experiment below measures exactly that. If base-plus-restoration recovers little over base-only, the project stops.

Why not just upscale the whole rendered frame with a fast network or lookup table? MGA reports that uniform computation wastes up to 48% of FLOPs. SplatSuRe shows that uniform super-resolution is worse and less view-consistent than selective super-resolution guided by geometry. MuLUT's roughly 240 ms per HD image on a phone CPU also makes full-frame lookup-table restoration too slow for a headset. We still run this baseline at an equal millisecond budget, because at small upscaling factors on a GPU it may be competitive.

Why not just stream 2D foveated video and restore it, as in CRFP? Pre-rendered video cannot provide motion parallax for free translation, as argued in the L3GS and adaptive-streaming papers. CRFP's 2D alignment is also not built for depth disocclusion. We do not run a full video-streaming comparison. We do test the narrower claim by comparing depth reprojection against 2D-flow alignment on translation-heavy segments of the traces.

Why not just run a scheduler and a restorer independently? This is the real threat. If a restorer placed after an unchanged L3GS scheduler captures nearly all the gain, the joint scheduler is unnecessary complexity. The decoupled ablation settles it directly. The prediction that would support our approach is that the decoupled pipeline wastes bits on objects the restorer would have fixed anyway, such as large smooth surfaces, while under-downloading thin structures. That should appear as a gap concentrated in objects with low restorability scores. If the gap is under a few tenths of a dB across scenes, we will report the decoupled design as the recommendation.

Why not just bake super-resolution into the streamed splats on the server, as SplatSuRe does? Baking improves the content being sent but does not reduce what must be sent. SRGS-style densification increases primitive count. We include a SplatSuRe-fitted scene streamed under L3GS as an extra condition, to check whether better source content alone closes the gap at low bitrates.

Proposed Experiments

We use Mip-NeRF 360, Tanks & Temples, and Deep Blending scenes, each fitted with L3GS-style base and enhancement layers. Viewer motion comes from the Quest 3 6DoF traces used by L3GS, and bandwidth comes from 5G throughput traces of the kind used there. Fidelity is PSNR, SSIM, and LPIPS against full-quality renders along each trajectory. Flicker is the error between consecutive frames after depth-based warping. System metrics are stall rate, per-frame client milliseconds, and energy, all measured on a Quest 3. For reference, Gong et al. report 126 FPS for splat rendering on that headset, which bounds the time left for restoration.

Three checks decide the project. The first is whether a restorer of a few NAFNet blocks, run on base-layer-only pixels within about 3–5 ms per frame, recovers a meaningful share of the gap between base and full quality. The second is whether the offline restorability score actually ranks objects by how much restoration helps them on held-out views, measured by rank correlation. The third, the main result, is whether joint scheduling beats the decoupled pipeline at equal bits and equal milliseconds on the trace-driven evaluation. We also sweep the compute budget downward to emulate thermal throttling and check that the joint scheduler degrades gracefully toward the bandwidth-only schedule.