Rate as a Permanence Test for Compact Gaussian Maps

Robots, phones and headsets increasingly want a photorealistic map of a place that is both small enough to store or stream and free of the people, cars and clutter that happened to pass through during capture. These two goals are handled by two separate literatures. One makes Gaussian-splatting scenes (3D Gaussian Splatting, or 3DGS, represents a scene as millions of small colored, semi-transparent ellipsoids that are projected and blended into images) far smaller by entropy coding. The other strips out transient "distractors". Neither looks at the other. Compressors assume that every training image shows the same static world, so they faithfully encode the floating debris left behind by passers-by. Distractor removers output hundreds of megabytes of raw, uncompressed primitives.

This note proposes treating these as one decision, made with one quantity: bits. Something that truly belongs in the permanent map explains many images for a single storage cost. A transient explains only the one or two images it appears in. So if the map must pay storage for every Gaussian it keeps, while a separate throwaway channel may explain each training image's leftovers for free at storage time (but at a per-image cost during training), then the rate penalty itself becomes a test of permanence. The catch is that a naive version of this test also throws away real but rarely seen structure, such as distant façades or corners seen from only a few cameras. Correcting for that uneven coverage is the central technical contribution.

Core Claim

A single rate–distortion objective can remove distractors and compress the map at the same time. It needs two parts. The first is a stored static Gaussian set whose entropy-coded bits are penalized. The second is a per-view distractor channel that is discarded before encoding. The penalty on that channel must be normalized by how many training cameras geometrically cover each region, not by how many agree on it. Joint training of this kind should beat a pipeline that first cleans the scene and then compresses it, measured as clean-view quality at equal file size. It should also do so without the uncorrected method's failure mode of deleting sparsely observed static geometry.

Mechanism

We start from HAC++, which stores anchors (seed points that each spawn a few Gaussians) and predicts the probability of each anchor attribute from a compact hash grid for arithmetic coding. It trains with a differentiable rate term and reports over 100× compression on static scenes. We use its rate model unchanged. To this we add the per-view camera-plane distractor Gaussians of DeSplat. DeSplat composites them in front of the static render and discourages them with an L1 opacity penalty. In our design they are never written to the bitstream. The objective is

$$\mathcal{L} = \sum_{v} D\big(I_v, \hat I_v(\mathcal{G}_s, \mathcal{G}_d^v)\big) + \lambda\, R(\mathcal{G}_s) + \beta \sum_{v} \sum_{p} w_v(p)\, \alpha_d^v(p),$$

where $I_v$ is training image $v$, $\hat I_v$ its render from static Gaussians $\mathcal{G}_s$ and that view's distractor Gaussians $\mathcal{G}_d^v$, $D$ is the usual L1+SSIM photometric loss, $R$ is the HAC++ estimated bit cost of the static anchors, $\alpha_d^v(p)$ is distractor opacity at pixel $p$, $\lambda$ and $\beta$ trade off rate against the distractor penalty, and $w_v(p)$ is a coverage weight described below.

A back-of-envelope reading shows why this rejects transients. Suppose some content appears in $k$ training views. Explaining it statically costs roughly $\lambda r$ once, where $r$ is its bit cost. Explaining it as a distractor costs about $\beta a$ in each of the $k$ views, where $a$ is the opacity mass it needs per view. It therefore enters the stored map only when $k\beta a > \lambda r$. That is a minimum-support threshold, and $\lambda$ now controls both file size and how strict the permanence test is.

The same arithmetic exposes the flaw. A static region covered by only three cameras has $k=3$ and fails the test just as a pedestrian does. This is the density irregularity familiar from point clouds, where distant structure receives little sampling support, and here it reappears as uneven view coverage. The fix is to set $w_v(p) = K/k_{\text{cov}}(x_p)$. Here $x_p$ is the 3D point where pixel $p$'s ray hits the current static surface (from the rendered static depth), $k_{\text{cov}}$ is the number of training cameras whose view frustums contain $x_p$ without depth-test occlusion by the static map, and $K$ is the median coverage over the scene.

After this change the threshold compares the fraction of covering views that support the content against a constant, rather than an absolute count. A pedestrian seen in 1 of 30 covering views has fraction 1/30 and is rejected. A far façade seen in 3 of 3 covering views has fraction 1 and is kept. Coverage is computed from camera geometry and the static depth alone, so a transient cannot inflate its own support.

We also borrow the key observation of RobustSplat: transients leave permanent floaters mainly by triggering densification, the step where the optimizer adds new Gaussians wherever image error is high. Accordingly, the image-space gradients that drive densification are weighted by $(1-\alpha_d^v)$, so pixels the distractor channel already explains cannot spawn stored Gaussians.

Research Program

The work has three parts. The first is building the joint model on the HAC++ codebase with DeSplat-style distractor planes. The second is an analysis of whether the rate term alone suppresses or encourages transient floaters. The file flags this open tension: HAC++ reports entropy loss acting as a regularizer against high-frequency noise, whereas RobustSplat attributes floaters to unregularized early fitting. The third is a stratified evaluation that bins test pixels by coverage count and by specularity, so that the irregularity correction and the known confusion between shiny static surfaces and motion are measured directly rather than averaged away.

Two failure modes are expected, and each has a planned response. Specular floors could be rejected as transients. We will test whether routing view-dependent color into a cheap shared predictor, as in the DC/AC factorization of MEGA, lowers their rate cost enough to keep them. Surfaces hidden in every view remain holes. That is out of scope and will be reported, not fixed.

Baselines

The naive reference point is HAC++ trained directly on the distractor-laden images, with no robustness mechanism. It shows how much transient debris a compressor encodes by default. It also answers whether entropy regularization already does our job. For context, HAC++ reports compressing a 1350MB Mip-NeRF 360 scene to 11.02MB, a more than 100× reduction, on clean static data. No reported result was found on Lacuna for HAC++ on the RobustNeRF or On-the-go datasets.

The strongest competitor is a two-stage pipeline. It first trains a distractor-free model, then compresses it with HAC++ using the first stage's transient masks to exclude those pixels from the compressor's loss. For the first stage we use RobustSplat, with 3DGS-HPC as an alternative since it combines patch-wise residual statistics with DINOv2 features. This baseline has access to the best available cleaning and the best available compression, so a joint method must beat it to justify itself. On distractor removal, the file's quantitative anchors are the RobustNeRF paper's 30.87 dB PSNR on its own dataset (versus 25.22 dB for Mip-NeRF 360) and 3DGS-HPC's reported 1.39–5.54 dB gain over plain 3DGS. RobustSplat claims best results on On-the-go but no exact number is recorded in the file.

For fairness, every method uses the same COLMAP initialization, the same 30k-iteration schedule and GPU, and an equal sweep of eight settings of its rate weight. We compare full rate–distortion curves. Beating the two-stage pipeline means at least 0.5 dB higher clean-view PSNR at matched megabytes over most of the curve, or equal PSNR at 25% fewer bytes, with no loss in the lowest-coverage bin.

Three ablations isolate the ingredients. The first removes the coverage weight ($w_v \equiv 1$) to show the irregularity bias. The second removes the rate term from the distractor decision, which reduces the method to DeSplat's opacity penalty followed by compression. The third removes the gated densification. An upper reference, HAC++ trained on the clean versions of composited scenes, bounds what is achievable.

Why not just…?

Why not just clean first with RobustSplat and then compress with HAC++? This is the real threat, and it may well come close. Our argument for joint training is that a cleaner makes its keep-or-drop decisions blind to cost. Any residual floater it keeps is paid for in bits, and ambiguous low-support content cannot be vetoed on cost grounds. That argument is untested. The matched rate–distortion comparison above settles it. If the two-stage curve matches ours, the honest conclusion is that the cost-based permanence test adds nothing over sequential tools, and the coverage-normalized analysis becomes the main contribution.

Why not just mask people and cars with a 2D segmenter before training? Semantic masks miss shadows, moved furniture and reflections, as 3DGS-HPC documents. Tracking-based 2D segmentation is also brittle under fast interaction: Split4D reports 0.592 mIoU for the tracking-based SA4D versus 0.893 for its native 3D clustering on Multi-Human. We will include this baseline (segmenter masks fed to HAC++) and report results on On-the-go scenes with non-semantic transients.

Why not just train vanilla 3DGS and prune low-opacity or rarely visible Gaussians afterwards? RobustSplat's analysis indicates that transient floaters are created by densification and are often opaque, so opacity pruning misses them. Visibility-count pruning is exactly the uncorrected rule that deletes sparsely seen static structure. Our coverage-binned evaluation measures both failures directly.

Why not rely on HAC++'s entropy loss alone? It may partly suppress floaters, since the rate penalty discourages noise. Without a free per-view channel, however, a transient that causes large image error still has nowhere else to go except stored Gaussians. The naive baseline quantifies this.

Proposed Experiments

The main benchmarks are the RobustNeRF dataset and the NeRF On-the-go dataset. Both provide held-out clean test views. We report PSNR, SSIM and LPIPS against stored megabytes. We also count floaters, meaning stored Gaussians more than a set distance from the depth surface rendered from clean views.

For controlled tests we composite COCO objects into Mip-NeRF 360 scenes, in the style of the synthetic set of Distractor-free Generalizable 3DGS. This lets us vary distractor frequency and camera coverage independently, and lets clean-trained HAC++ serve as a true upper bound. Phototourism is reserved for a qualitative stress test, because its illumination changes add a confound our model does not address.

The decisive checks are three. First, the joint method beats the two-stage pipeline at matched size. Second, the coverage weight recovers lowest-coverage-bin PSNR that the unweighted variant loses, while keeping floater counts low. Third, the chosen $\lambda$ behaves as the derived support-fraction threshold predicts: on composites, the minimum distractor frequency that gets stored should scale with $\lambda$.

As a secondary downstream test, we run GSLoc-style render-and-compare pose refinement on the compressed maps. The queries are held-out On-the-go frames containing pedestrians. This measures whether the smaller, cleaner map also localizes more reliably than a compressed map that still contains encoded floaters.