3D Gaussian Splatting represents a scene as millions of small colored ellipsoids that are projected and blended into images. It renders very fast, but it is wasteful. When each ellipsoid also stores a high-dimensional semantic vector, such as a 512-number feature from a vision foundation model, the memory cost becomes severe: Feature 3DGS, which attaches distilled features to every Gaussian, needs about 1.17 GB per Replica scene. Semantics change far more slowly across space than color does, so most of those feature vectors repeat what their neighbors already hold. DF-3DGS shows the semantic field can get by with roughly 38k Gaussians against 910k for color. It gets there by splitting the scene into two unrelated point clouds, which discards the shared geometry.
We propose to keep a single cloud and learn a second, separate decision for each Gaussian. Beyond whether it exists at all, the model decides whether it carries a feature vector. Gaussians that carry no feature still shape color and geometry, but they are transparent to the feature renderer. The carrier decision is a differentiable gate trained from the feature reconstruction loss. A local diversity penalty, borrowed from determinantal point processes (probability models that favor sets of mutually dissimilar items), discourages nearby carriers from storing near-identical vectors. The prediction is concrete. About 5% of Gaussians should end up as carriers, which matches the sparsity DF-3DGS finds with a separate cloud. Semantic accuracy should match or exceed DF-3DGS, and the semantic loss should still be able to improve the shared geometry, as FHGS found when feature gradients drive Gaussian positions.
Core Claim
Existing work treats semantic redundancy in one of two ways. It either removes whole primitives, or it separates semantics into its own primitive set. Neither is quite right. The redundancy sits in the feature attribute, not in the primitive. The Gaussians needed to render a sharp texture are mostly not needed to say "this is a chair," but they are the same physical surface. We claim that factoring "exists" from "carries features" inside one geometry gives three things together. Feature storage reaches the sparsity of a decoupled field. Feature gradients can still correct geometry. And the feature vectors can stay fixed during optimization, avoiding the distortion FHGS attributes to backpropagating into them. That last point addresses a disagreement in the literature. FHGS freezes features and lets gradients reach only geometry and opacity. CF3 and the Gumbel-Softmax masks of FastAvatar keep everything differentiable. Our design makes the selection differentiable while the embeddings stay fixed.
Mechanism
Each Gaussian $i$ has a standard alpha-blending weight $w_i(p)$ at pixel $p$, a carrier gate $g_i \in [0,1]$, and a compact feature code $z_i$. The gate $g_i$ is relaxed with a hard-concrete or Gumbel-Softmax distribution during training, as in FastAvatar, and snapped to 0 or 1 at export. The code $z_i$ is a low-dimensional latent produced by a scene autoencoder trained on the 2D teacher features, following DF-3DGS's semantic compression. Its value is set by weighted multi-view fusion of the teacher features, as CF3 does, and is refreshed periodically rather than updated by gradients. The rendered feature is a blend renormalized over carriers only:
$$ \hat f(p) = \frac{\sum_i g_i\, w_i(p)\, z_i}{\epsilon + \sum_i g_i\, w_i(p)}, $$
Here $\epsilon$ is a small constant. The renormalization means a few carriers can paint a whole object region without dense neighbors. The training objective is
$$ \mathcal{L} = \mathcal{L}_{\text{rgb}} + \lambda_f \mathcal{L}_{\text{feat}} + \lambda_0 \sum_i g_i - \lambda_d \sum_{t} \log\det\!\left(K_{t} + \delta I\right). $$
$\mathcal{L}_{\text{feat}}$ compares $\hat f$ with the teacher feature map after the decoder. $\lambda_0$ prices each carrier. $t$ ranges over image tiles or octree cells. $K_t$ is a kernel matrix over the gated codes of carriers in that cell, combining feature similarity and spatial proximity. $\delta$ keeps the matrix well-conditioned.
The log-determinant penalizes groups of carriers that are close together and similar in feature, which is the repulsive effect that Priors for Diversity in Generative Latent Variable Models uses to stop latent variables from encoding the same thing. That paper also names the obstacle: a global determinant costs cubic time in the number of items, which is impossible for $10^5$ to $10^7$ Gaussians. Restricting the determinant to cells with at most a few dozen carriers keeps the cost linear in scene size. This is the methodological contribution borrowed from the feature-redundancy direction.
Two further roles are worth keeping straight. FHGS's attraction and repulsion along rays is the closest existing regularizer. It is a hand-tuned force on positions, not a selection prior, and it prunes nothing. Feature gradients also flow, through $w_i(p)$, into position and opacity, so semantics can help sharpen geometry, as FHGS reported.
Deciding which Gaussians exist is handled by the decoupled optimizer of A Step to Decouple Optimization in 3DGS, referred to below as AdamW-GS. It updates only visible Gaussians with a sparse version of the Adam optimizer, and its regularizers drive redundant Gaussians to zero opacity on their own, cutting 41–49% of primitives. Fixing this backbone keeps the comparison with photometric pruning methods fair.
Research Program
The first stage builds the gated, renormalized feature rasterizer as a small change to the 3DGS feature-rendering path, and verifies it reproduces Feature 3DGS when all gates are fixed at 1.
The second stage trains the full objective on Replica and DTU. Its main measurement is the carrier fraction, and how that fraction tracks semantic spatial frequency. We expect more carriers at object boundaries and thin structures, and fewer in large uniform regions.
The third stage tests the claimed advantage over decoupling. It asks whether the feature loss acting on shared geometry improves geometry, measured by Chamfer distance on DTU. It also asks whether semantic edges agree with color edges better than with two separate clouds, measured by boundary F-score between rendered segmentation and ground-truth boundaries on Replica.
The fourth stage scales to KITTI-360 to check that the local determinant stays cheap. It also measures total training time, because CF3's offline pipeline adds about 30 minutes per scene, and our method should add only a small fraction of baseline training time.
Baselines
The naive reference point is Feature 3DGS with its 128-dimensional rasterization and 1×1 convolutional decoder, with no pruning. DF-3DGS reports this baseline at 73.9% mIoU and 1.17 GB on Replica. Everything must improve on its memory without losing accuracy.
The strongest existing method on semantic accuracy and size is DF-3DGS. It trains a separate sparse semantic cloud with an adaptive codebook and a 512-to-9 dimensional autoencoder, and reports 77.9% mIoU, 13 MB of semantic storage, 36 FPS, and about 38k semantic Gaussians on Replica. We will run it from the same teacher features, the same autoencoder latent size, and the same iteration budget. Beating it means matching its mIoU within noise at equal or smaller total storage, counting color and semantics together, while also showing a measurable gain in Chamfer distance or boundary agreement that its decoupled design cannot provide.
CF3 is the strongest post-hoc compactor. It fuses multi-view features onto a pretrained 3DGS, uses a per-Gaussian autoencoder down to three dimensions, and prunes and merges by contribution and Mahalanobis distance, reaching as little as 5% of Feature-3DGS's Gaussians. The file has no CF3 mIoU number on Replica, so we will rerun it under our protocol and compare accuracy, size, and wall-clock time, where its roughly 30-minute overhead is the target to beat.
FHGS is the geometry reference. It reports a Chamfer distance of 0.75 on DTU against 1.96 for standard 3DGS, with 183 MB on scan 24. Our method should approach that geometry at a fraction of the memory.
For RGB fidelity on Mip-NeRF 360, the references are AdamW-GS, which our method uses as its backbone, and RadSplat, which reports SSIM 0.843 with 10× fewer points. Adding the feature machinery should not lower PSNR or SSIM relative to the backbone alone.
Three ablations isolate the key ingredients. The first keeps the carrier renderer but chooses carriers heuristically, by thresholding each Gaussian's feature-weighted ray contribution or by farthest-point sampling at the same carrier budget. The second uses learned gates with only the sparsity penalty $\lambda_0$, with the diversity term removed. The third lets gradients flow into $z_i$, to test FHGS's claim that this distorts features.
Why not just…?
Why not just decouple color and semantics, as DF-3DGS does? This is the strongest competitor on storage, and it may well tie us on mIoU. The case against it is structural. Its semantic Gaussians never touch the color geometry, so semantic supervision cannot fix geometry. FHGS's DTU result, 0.75 against 1.96 Chamfer distance, suggests that feedback is worth having. Our third stage tests this directly. If shared geometry yields no Chamfer or boundary improvement over DF-3DGS, our method's only remaining advantage is a single exported asset, and we will report it that way.
Why not just pick carriers with a heuristic after training? This is the real threat. Choosing the top few percent of Gaussians by feature-weighted ray contribution, in the style of RadSplat, may capture most of the benefit with no gates at all. The first ablation settles it at a matched carrier budget. If the heuristic matches the learned gates, the contribution narrows to the renormalized carrier renderer, and we would recommend the heuristic. We expect learned gates to win on thin classes and boundaries, where ray contribution is low but semantic information is high, and we will report mIoU on small Replica classes separately to check.
Why not just shrink the features with PCA and keep standard opacity pruning? Reducing dimension per Gaussian does nothing about the number of Gaussians storing redundant vectors. And Gula et al. show that the highest-variance directions of deep features often encode background and domain shifts rather than the target signal. We include a PCA-to-same-dimension variant of the latent as a control against the learned autoencoder.
Why not just use the sparsity penalty without the determinant term? An L0-style penalty alone may already give sparse carriers. The determinant term earns its place only if, at equal carrier count, it reduces the number of pixels with no carrier coverage and improves accuracy in sparsely viewed regions. That is where A Step to Decouple Optimization in 3DGS reports pruning artifacts. The second ablation decides this, and if the term does not help, we drop it.
Proposed Experiments
The main experiment uses Replica with LSeg features. It reports mIoU, semantic and total storage, carrier fraction, FPS, and training time against Feature 3DGS, DF-3DGS, and CF3. DTU tests geometry through Chamfer distance against FHGS and standard 3DGS. Mip-NeRF 360 and Tanks and Temples confirm that RGB PSNR and SSIM are preserved relative to the AdamW-GS backbone. Because those datasets lack semantic labels, feature fidelity there is measured as cosine similarity to held-out teacher feature maps. KITTI-360 tests scaling against CF3's reported 141.6 FPS and 6.2M memory figure.
Two checks decide the outcome. The first is whether learned carriers beat the heuristic carrier selection at equal budget on boundary and small-class mIoU. The second is whether the shared geometry improves Chamfer distance and boundary agreement over DF-3DGS at equal storage. If both fail, the idea reduces to a clean way of storing sparse features in one cloud. If both hold, the result is evidence that semantic redundancy should be pruned per attribute rather than per primitive.
References
- Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields
- Efficient Decoupled Feature 3D Gaussian Splatting via Hierarchical Compression
- FHGS: Feature-Homogenized Gaussian Splatting
- CF3: Compact and Fast 3D Feature Fields
- FastAvatar: Towards Unified and Fast 3D Avatar Reconstruction with Large Gaussian Reconstruction Transformers
- Priors for Diversity in Generative Latent Variable Models
- A Step to Decouple Optimization in 3DGS
- RadSplat: Radiance Field-Informed Gaussian Splatting for Robust Real- Time Rendering with 900+ FPS
- Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection