Click-to-Gaussians: Feed-Forward Promptable 3D Segmentation Trained from Sparse Labels

Selecting one object in a 3D scene, such as clicking a chair in a photo and getting that chair in 3D, currently requires two slow steps. First, a scene model is fitted to the captured images by thousands of gradient steps. Second, a segmentation layer is fitted or graph-cut on top of that model. The scene models are typically 3D Gaussian Splatting: the scene is represented as many small colored, semi-transparent blobs that are projected onto the image to render it. The whole pipeline costs minutes per scene. A separate line of work now predicts those Gaussians directly from a few photos in one network pass. These feed-forward predictors only output appearance or fixed text-matched semantics, and none accepts a user's click.

We propose a feed-forward network that takes a handful of posed images plus a click or scribble on one of them and returns, in well under a second, a set of Gaussians with the selected object marked. The less obvious part is how it is trained. We supervise it only with sparse labels, meaning a few clicks or scribbles per object, rather than dense masks. The predicted Gaussians themselves carry those labels across views: a Gaussian that a scribble touches in view A is the same Gaussian that renders into views B and C. The 3D representation is therefore both the output and the mechanism that turns sparse labels into dense ones.

Core Claim

A generalizable Gaussian predictor with a per-Gaussian affinity embedding can be trained on many scenes using only sparse 2D labels. The training loss is applied to masks rendered from the Gaussians rather than to per-view 2D predictions. At test time it can segment objects in unseen scenes from a single click, with no per-scene optimization, at accuracy close to methods that optimize per scene. Sparse labels suffice because every rendered view shares the same Gaussians, so each label constrains all views at once. Dense 2D teacher masks, by contrast, add boundary detail but also add the view-to-view disagreement documented by CaRF and G²TAM.

How this relates to prior work

Three methods motivate the speed gap, because they reach high click-based accuracy only after optimizing per scene. Segment Any 3D Gaussians (SAGA) distills multi-scale SAM masks into per-Gaussian features with per-scene contrastive training and then answers prompts in 4 ms. GaussianCut lifts video-tracked masks onto an optimized 3D Gaussian scene and runs a graph cut, taking about 90 seconds per scene. NVOS and SPIn-NeRF do the analogous thing for NeRF, a neural scene representation, with a per-scene network and a graph cut. We keep SAGA's idea of a per-Gaussian affinity feature that is queried at click time. We predict that feature feed-forward instead of distilling it per scene.

Three feed-forward systems supply the method or constrain it. FreeSplat supplies the geometry backbone: indoor feed-forward Gaussians from long sequences. It also supplies a constraint, since memory exceeds 40 GB beyond about 50 images, which caps our view count. SegSplat and GSemSplat predict semantic Gaussians feed-forward. Both answer only text queries, and both supervise on per-view SAM or CLIP outputs. GSemSplat documents the resulting trade-off: SAM features give sharp boundaries but misidentify regions, while CLIP-style features give context but blurry edges. Our task is interactive selection, and our supervision is sparse labels applied through rendering.

G²TAM is the closest promptable feed-forward system. It fuses click, box, or text prompts with image tokens early in the encoder and outputs 3D point maps plus 2D instance masks. It produces no renderable Gaussians and is trained on dense instance labels. Its ablation constrains our design: injecting the prompt late cost more than 10 S-mIoU (Spatial mIoU, the benchmark's 3D-consistent overlap score), dropping to 61.8 from 72.3 for early fusion.

Rewis3d supplies the core supervisory insight. It shows that feed-forward 3D reconstruction can carry scribbles across video frames well enough to close about 90% of the gap to dense supervision on Waymo. It even beats real LiDAR depth for this purpose, because the reconstruction is dense and reports per-point confidence. Rewis3d throws the 3D away after training a 2D segmenter. We keep the 3D and make it the product.

Research Program

The model starts from a FreeSplat-style predictor that outputs Gaussian positions, shapes, opacities, and colors from $N \le 16$ posed views. We add a head that gives each Gaussian $i$ a low-dimensional affinity embedding $f_i$. Clicks and scribbles are encoded as prompt tokens and fused into the multi-view encoder, following G²TAM's early-fusion result. A light prompt-conditioned head then outputs a per-Gaussian foreground probability $p_i$. Rendering $p_i$ with the same projection and blending used for color gives a soft mask $M_v$ in any view $v$. The prompt-agnostic embeddings $f_i$ also allow a cheap path: a new click selects Gaussians by embedding similarity without re-running the encoder. We compare that path against re-running with early fusion.

Training uses ScanNet++ scenes from the InsTrack split of 856 training scenes. These scenes have dense instance labels, which lets us simulate sparse annotation and also train a dense-label upper bound. Each training episode samples a scene, a view subset, and an object. It places one prompt click in a randomly chosen view and draws a few scribble pixels $S_v^{+}$ (on the object) and $S_v^{-}$ (off the object) in other views. The loss is

$$ \mathcal{L} = \sum_{v} \sum_{x \in S_v^{+} \cup S_v^{-}} \mathrm{BCE}\big(M_v(x), y_v(x)\big) \;+\; \lambda \sum_{(i,j) \in \mathcal{N}} w_{ij}\,(p_i - p_j)^2 , $$

where $y_v(x)$ is the scribble label at pixel $x$. The second term is a smoothness penalty over pairs $(i,j)$ of nearby Gaussians in a $k$-nearest-neighbor graph $\mathcal{N}$, which pushes neighbors toward the same label. Its weights $w_{ij}$ fall off with color and position differences, as in GaussianCut's pairwise term. Here the smoothness is a training-time loss, so at inference no graph cut is needed. Following Rewis3d, both terms are down-weighted where the predictor's geometric confidence is low. The embedding head receives an additional contrastive loss that pulls together Gaussians labeled as the same object and pushes apart different objects, using only scribble-touched Gaussians.

Three supervision regimes are trained under identical schedules: sparse scribbles only, per-view SAM pseudo-masks only, and both together. Their comparison is the scientific core of the project.

Baselines

The naive reference point is per-view lifting. The user's click is tracked through the views with SAM2, and each 2D mask is projected onto the same feed-forward Gaussians by accumulating projection weights, the lifting step GaussianCut uses before its graph cut. Nothing is learned in 3D. Any gain we report must exceed this, run on the same views and the same predicted Gaussians.

The strongest per-scene methods are SAGA and GaussianCut, run with their released settings on fully optimized 3D Gaussian scenes. On the NVOS benchmark (LLFF scenes), SAGA reports 92.6% mIoU and GaussianCut 92.5% mIoU. On the SPIn-NeRF dataset, GaussianCut reports 92.9% mIoU. They are the strongest competitors because they set the accuracy ceiling. We do not expect to exceed it with a few views and no optimization. Beating them on the speed–accuracy trade-off means reaching within about 2 mIoU of these numbers at under one second per prompt including reconstruction, against their minutes. For fairness, we also run them restricted to the same $N$ input views our model sees, where their accuracy may drop.

The strongest feed-forward promptable competitor is G²TAM, which reports 74.3 S-mIoU on InsTrack, against 47.6 for SAM2. We compare on the 50 InsTrack validation scenes using the same prompts. Beating it means matching or exceeding 74.3 S-mIoU while also producing renderable segmented Gaussians.

SegSplat cannot take clicks, but it can be adapted by selecting the cluster that contains the clicked Gaussian. This tests whether a fixed semantic memory bank is enough for interactive selection.

Two ablations isolate the key ingredient. The first is the dense-label model trained on full ScanNet++ masks, an upper bound that measures how much of the sparse-to-dense gap is closed. The second is a version trained with the same sparse labels applied directly to 2D per-view predictions, not through rendered Gaussians. Its gap to our model measures the value of propagating labels through shared 3D Gaussians.

Why not just…?

Why not just track the click with SAM2 and lift the masks onto feed-forward Gaussians? This is the most plausible threat, and it is our naive baseline. Prior evidence predicts it fails under wide view changes. G²TAM measures SAM2 at 47.6 S-mIoU against 74.3 for geometry-grounded fusion. CaRF and GaussianCut both report flickering and leakage when per-view masks are lifted. Our feed-forward setting uses few, widely spaced views, which is exactly that regime. If lifting matches us on the closely spaced LLFF frames of NVOS, we will report that, and we will restrict the claim to wide-baseline captures such as InsTrack.

Why not just run GaussianCut's graph cut on feed-forward Gaussians? Feed-forward models produce far fewer Gaussians; FreeSplat removes about 55% of redundant ones. A graph cut on them could take seconds rather than 90 seconds, which would remove most of our speed argument. We therefore include it as a baseline. If it matches our accuracy at comparable latency, the contribution narrows to the training finding: that learned affinities from sparse labels improve the graph's unary inputs, the per-Gaussian foreground scores, over lifted SAM2 masks. We would test that by feeding our $p_i$ into the cut.

Why not just distill dense SAM masks, as SegSplat and GSemSplat do, instead of using sparse labels? SAM masks are free, whereas scribbles require annotators. GSemSplat's documented trade-off between sharp boundaries and correct context, and CaRF's cross-view inconsistency, suggest that dense-but-inconsistent teachers cap quality. The three-regime comparison settles this directly. If SAM-only training wins, the paper's claim becomes "rendering-based lifting of SAM," and sparse labels become an optional consistency term.

Why not just use G²TAM and attach its masks to Gaussians? G²TAM's point maps lack absolute scale and are not renderable, so it cannot produce a novel view of the segmented object. We test whether its masks, lifted onto FreeSplat Gaussians, rival our end-to-end model on held-out-view mIoU.

Proposed Experiments

On InsTrack we report S-mIoU and per-prompt latency against all baselines. On NVOS and the SPIn-NeRF dataset we report mIoU on held-out views, evaluated zero-shot because these scenes are not in training, using 8 and 16 input views. Under the matched-view protocol, per-scene methods receive the same views. We vary the scribble budget from a single click to dense masks and plot mIoU against annotation cost. This shows how much of the gap to dense labels is closed: Rewis3d closed about 90% in 2D on Waymo, and we ask whether a 3D output reaches similar levels.

The decisive checks are three. The first is whether rendering-based sparse training beats the per-view-loss ablation by a clear margin on wide-baseline InsTrack scenes. The second is whether the full model beats per-view lifting and the graph cut on feed-forward Gaussians at under one second. The third is whether early prompt fusion is needed, by comparing re-encoding with the prompt against querying the fixed embeddings $f_i$. If the first check fails, the shared-Gaussian propagation idea is wrong for this setting. If the second fails, the feed-forward speed claim reduces to an engineering result.

We name one expected failure in advance. Textureless walls and specular surfaces break the cost-volume geometry that FreeSplat relies on, and errors there will spread labels through bad Gaussians. Confidence weighting is our only defense, so we report results separately on those scene regions.