One Tree for Detail and Meaning: Scale-Queryable Semantic Gaussians That Render Faster When You Ask Less

When a 3D scene is reconstructed from photos with Gaussian splatting (the scene is stored as millions of small, colored, semi-transparent 3D blobs that are projected onto the screen and blended), it can also store "meaning." Features from 2D vision models such as CLIP or SAM are copied onto each blob, so the scene can answer questions like "where is the mug?" These semantic scenes have two separate problems. The first is that they answer at one fixed granularity: a user cannot ask for "the bolt," then "the bucket," then "the excavator" with a single dial. The second is that they slow down sharply as features are added. Both problems have the same cause. Every blob is treated as a peer, with no notion of which blobs are parts of which larger things.

We propose to store the scene as a single tree of Gaussians. Leaves carry fine color detail. Internal nodes are merged parent Gaussians that carry the semantic features for progressively larger physical extents. A query at physical scale $s$ ("things about 5 cm across" versus "things about 1 m across") then simply selects a cut through the tree, and the same mechanism selects a cut for rendering when a view is distant. The result is a continuous scale-conditioned feature field that runs at rasterization speed, because asking a coarse question means drawing fewer, larger blobs. Coarse groups contain fine groups by construction rather than through a soft penalty.

Core Claim

Two existing ideas can be unified into one structure. The first is scale-conditioned grouping in the style of GARField, which learns an affinity field $F_g(x,s)$ over 3D position and physical scale inside a slow neural radiance field (a network queried hundreds of times per pixel). The second is geometric level of detail for splatting in the style of Multi-Scale 3DGS, which merges small Gaussians into coarser parents and swaps them in when a child covers fewer than two pixels.

The claim has two parts. The parent nodes of a level-of-detail tree are the right place to store coarse semantics. And if the tree is built using cross-modal evidence, meaning SAM mask co-membership as well as spatial proximity, then a single hierarchy delivers three things at once. It delivers the sparse semantic field that DF-3DGS obtains with a separate, hand-decoupled 38k-Gaussian cloud. It delivers continuous scale queries without a per-primitive network. And its semantic rendering cost falls as the query scale or viewing distance grows.

The tree also removes a failure that nobody has measured yet. When level-of-detail culling is applied to a feature-carrying Gaussian set, fine primitives are dropped based on visual footprint alone. Their features vanish with them, and semantic boundaries corrupt at intermediate distances (Gap 2 in our notes, arising from the opacity manipulation in CLoD-GS and FHGS). In a shared tree, a culled child is always replaced by a parent that holds the semantic summary of that child.

Mechanism

The system starts from a trained color 3DGS scene. It then builds a binary-ish tree bottom-up by repeatedly merging pairs of nearby Gaussians. The merge cost combines spatial overlap with disagreement in the SAM masks the two Gaussians project into across training views. This means parents do not straddle object boundaries, which matters for semantics and also addresses the axis-aligned voxel rigidity of Multi-Scale 3DGS.

Each node $n$ has a physical radius $r_n$ (three standard deviations of the merged Gaussian) and a low-dimensional feature $f_n$. A shared small decoder lifts $f_n$ to the teacher's dimension, as in the speed-up module of Feature 3DGS and the 9-dimensional latent of DF-3DGS. Each node's semantic feature is visible over a band of scales, with a smooth fade between a node and its parent $p(n)$:

$$w_n(s) = \mathrm{clip}\!\left(\frac{\log(r_{p(n)}/s)}{\log(r_{p(n)}/r_n)},\,0,\,1\right),$$

Here $w_n(s)$ is the fraction of the node's opacity used when rendering features at query scale $s$. The parent receives the complementary weight. Nodes with $w_n(s)=0$ are skipped entirely.

The active set for a semantic render is the coarser of two cuts. One is the cut implied by $s$. The other is the screen-coverage cut that would be used for color. The number of rasterized semantic primitives therefore shrinks monotonically as $s$ or camera distance grows.

Training follows GARField's recipe of contrastive supervision from multi-scale SAM masks lifted to metric scale, applied to rendered cut features instead of NeRF ray samples. The feature distillation follows Feature 3DGS. FHGS's observation constrains the design: semantic features are view-independent, and letting appearance gradients reach them corrupts them. Node features are therefore isotropic (no spherical harmonics), and color and semantics share geometry but not attributes.

Hierarchical grouping becomes reading groups off the cut. Groups at a larger $s$ are unions of groups at a smaller $s$, because each coarse node is literally the ancestor of the fine nodes. This replaces GARField's greedy, post-hoc recursive clustering and its containment loss with a structural guarantee.

Research Program

The work has three parts.

Tree construction. We will compare merge criteria: purely spatial merging versus spatial plus mask merging. For each, we measure how often a node's descendants span multiple ground-truth instances.

Distillation and rasterizer. We will add scale-conditioned feature distillation on the tree and extend the tile rasterizer to evaluate $w_n(s)$ and skip inactive nodes. Color and feature passes share one depth sort over the union cut.

Analysis. We will characterize semantic rendering cost as a function of $s$ and of camera distance, and measure feature consistency as the camera zooms out. Consistency is defined as agreement between labels rendered from far views and labels rendered from near views of the same surface points, reprojected using known depth. We expect the tree to hold this agreement where independently culled baselines lose it.

Baselines

Naive reference. The "do nothing" point is projecting 2D SAM masks or CLIP features straight into 3D without optimization. GARField reports this yields 17.4 mIoU at the fine level on its bouquet scene, against 76.0 for its learned field. We will rerun it on all scenes so every method shares one number grid.

Hierarchical quality. The strongest competitor here is GARField itself, which defines the task and reports the 76.0 fine-level bouquet mIoU above. It is the bar on its five scenes (teatime, bouquet, keyboard, ramen, living_room) at fine, medium and coarse levels. We will run its released configuration on the same images and SAM masks. Beating it means matching its mIoU within one point at every level while rendering at interactive rates; its reported cost is about 20 minutes of training after NeRF pre-training and far-below-real-time querying.

Discrete hierarchy in splatting. Hi-LSplat builds a whole/part/sub-part tree with contrastive losses. It reports gains of up to 34.14 mIoU on ScanNet and 8.0 mIoU on LERF over other methods, plus the best Hierarchical Consistency on Hi-LERF and Hi-3DOVS. No absolute Hierarchical Consistency value was found on Lacuna. We will train it on identical splits. Beating it means equal or better Hierarchical Consistency at all three levels, together with lower frame time at coarse levels, which Hi-LSplat cannot provide because its hierarchy does not prune rendering.

Efficiency. The strongest efficient semantic splatting system is DF-3DGS. On Replica it reports 77.9% mIoU, 36 FPS, 13 MB of semantic storage, and about 38k semantic Gaussians. We match its feature teacher, latent size, iteration budget and GPU. Beating it means at least 77.9% Replica mIoU at single-scale queries with comparable storage, plus a rising FPS curve for coarse queries and distant cameras, where DF-3DGS stays flat.

Bolt-on composite. This is the most dangerous baseline. It takes DF-3DGS's semantic field with three discrete per-node scale features and pairs it with an independent level-of-detail color field built by CLoD-GS or Multi-Scale 3DGS. It gets the same tuning budget as our method.

Ablations. These isolate the key ingredients. The first builds the tree with spatial-only merging, testing whether the cross-modal merge criterion matters. The second replaces structural containment with GARField's containment loss on a flat set. The third replaces the continuous fade with hard level switching, and is scored on temporal flicker during zooms.

Why not just…?

Why not just store three discrete scale features per Gaussian and compress them, as Hi-LSplat and DF-3DGS make easy? This is a real threat on query accuracy. With 9-dimensional latents on a sparse semantic cloud, three levels cost little memory, and interpolation between levels may answer most queries well. Its weaknesses are elsewhere. It renders every semantic primitive regardless of scale or distance. Its coarse groups are not guaranteed to contain its fine groups. And it says nothing about what happens when the color field is culled. The bolt-on composite baseline is exactly this design. If it matches our method on mIoU, Hierarchical Consistency, zoom consistency and frame time, the shared-tree claim fails and we will report that.

Why not just splat a scale-agnostic feature and decode it per pixel with a small network conditioned on $s$? This extends Feature 3DGS's screen-space decoder. Per-pixel decoding at 1080p is cheap, so for 2D query maps this could rival our method, and we include it as a baseline. Its limitation is that features from different depths are alpha-blended before scale is applied. Scale-specific 3D groups therefore do not exist, and 3D selection or editing at a chosen scale again needs post-hoc clustering. We will test this directly with 3D group extraction accuracy and with containment violations measured across scales.

Why not just cluster Feature 3DGS or DF-3DGS features with recursive HDBSCAN, as GARField does? GARField's own pipeline shows this is greedy, non-differentiable, and prone to spurious fine clusters. It also inherits the flat rendering cost. We run it as an extraction baseline on the GARField scenes and report fine-level mIoU and violations of the containment property.

Why not just apply an existing level-of-detail method to Feature 3DGS? The pruning criteria in CLoD-GS and Multi-Scale 3DGS are purely radiometric, so fine feature-carrying primitives disappear without a semantic stand-in. The zoom-consistency experiment measures exactly this drop, and it is the effect our tree is designed to remove.

Proposed Experiments

Semantic accuracy will be measured on Replica, whose hierarchical "Segmentation Forest" also provides multi-level ground truth, and on ScanNet. Open-vocabulary localization will use LERF scenes, against LERF's reported 80.3% localization accuracy, and Hi-LERF/Hi-3DOVS. Multi-granularity grouping will use GARField's five scenes. To confirm color quality is not traded away, we report multi-scale PSNR and FPS on Mip-NeRF 360 and Tanks and Temples against Multi-Scale 3DGS, which reports 160% to 2400% speedups at 4× to 128× downscaling.

Speed versus question size. The first decisive check plots semantic frame time against query scale $s$ and camera distance on Replica and the GARField scenes. Our prediction is a monotonically falling curve. Flat-field baselines should stay constant near DF-3DGS's 36 FPS regime.

Zoom consistency. The second check runs a camera-pullback protocol. Near-view labels are reprojected to far views, and we measure retained mIoU. We expect independently culled baselines to lose fine-part labels at intermediate distances, while our method degrades gracefully to parent labels.

Accuracy parity. The third check is fine/medium/coarse mIoU and Hierarchical Consistency against GARField and Hi-LSplat. The method is worth pursuing only if it stays within one point of the best of them while delivering the speed curve.

All methods use the same SAM and CLIP teachers, poses, training-iteration budget and single GPU. Frame times are measured at matched resolution, excluding I/O.