A popular way to find objects inside a 3D scene reconstruction is to let every surface element "vote" for the center of the object it belongs to. Each element predicts an arrow pointing from itself to where it thinks the object's middle is, and elements whose arrows land in the same spot are grouped as one object. When the scene is reconstructed from photos with 3D Gaussian Splatting (3DGS, which represents a scene as millions of small colored, semi-transparent blobs), there is no 3D ground truth to train these arrows. The current approach, VoteSplat, supervises them with the centers of 2D object masks produced by the Segment Anything Model (SAM) in each training image. That supervision is systematically wrong in two predictable situations. The first is an object cut off by the image border, whose 2D mask center is not the projection of its 3D center. The second is a textureless boundary, such as a white cabinet against a white wall, where SAM's mask leaks across objects or merges them. Because supervision flows only from 2D to 3D, these errors pass straight into the 3D arrows.
We propose to reverse part of that flow. The current 3D votes, clustered and re-rendered into every view, serve as an independent check on each 2D mask. Each (object, view) pair gets a reliability weight based on whether the mask is truncated and whether it agrees with what the 3D reconstruction predicts. Masks that disagree are down-weighted, and in textureless regions they are split along rendered depth discontinuities. The claim is narrow and testable: most of the error in 2D-supervised 3D voting comes from a small, detectable subset of bad (object, view) masks, and removing their influence recovers accurate centers and instances without a better 2D segmenter.
Core Claim
The error chain from 2D to 3D has been documented across many pipelines. Lifting stages cannot recover information that the upstream detector discarded (View Invariant 3D Human Pose Estimation). Monocular depth errors pass directly into signed-distance and Gaussian geometry (MonoSDF, DN-Splatter). VoteSplat itself reports failures when object footprints exceed the camera's field of view and when objects sit inside concave containers. We add a specific observation about voting. Multi-view reconstruction gives every object many redundant 2D observations, and the 3D votes can predict what each of those observations should look like. That makes it possible to audit the 2D evidence per view, which a single-image pipeline cannot do. Two earlier works supply the ingredients. Accurate Localization of 3D Objects from RGB-D Data Using Segmentation Hypotheses scored multiple 2D foreground hypotheses with a 3D model instead of trusting one 2D box. UPIDet showed that sending 3D information back into the 2D branch helps cross-modal detection. Neither was applied to self-supervised voting in a radiance representation. Kim et al. use handcrafted part models on single RGB-D frames, and UPIDet works on LiDAR boxes with ground-truth labels. We use the reconstruction's own votes as the auditor, across views, with no 3D labels.
Research Program
We build on a geometry-regularized 3DGS backbone, DN-Splatter-style depth and normal priors, so that rendered depth is reliable on blank walls. Each Gaussian $i$ with position $x_i$ carries a learnable offset $o_i$, as in VoteSplat. For each instance $k$ we keep an explicit 3D center $c_k$. SAM gives mask $m_{k,v}$ with 2D centroid $\mu_{k,v}$ in view $v$, and $\pi_v$ is that view's camera projection. The voting objective is
$$ \mathcal{L}_{\text{vote}}=\sum_i \rho\big(x_i+o_i-c_{k(i)}\big)+\lambda\sum_{k,v} w_{k,v}\,\rho\big(\pi_v(c_k)-\mu_{k,v}\big), $$
where $\rho$ is a robust (Huber) penalty, $k(i)$ is the instance currently assigned to Gaussian $i$, and $w_{k,v}\in[0,1]$ is the per-view reliability of the 2D evidence. With $w_{k,v}\equiv 1$ and the center constraint folded into per-Gaussian targets, this reduces to a VoteSplat-like objective. That reduction is the ablation that isolates our contribution.
The reliability weight combines two checks. The truncation check sets $w_{k,v}$ low when the rendered footprint of instance $k$, computed from the current vote cluster, extends past the image border by more than a set fraction. In that case the 2D centroid is biased by construction. The agreement check compares $m_{k,v}$ with the rendered mask of cluster $k$ in view $v$ using intersection-over-union. It additionally flags masks that cross a strong rendered depth discontinuity, because on textureless surfaces geometry separates objects that color cannot. When a flagged SAM mask covers two depth-separated clusters, we split it at the depth edge instead of discarding it. This is the step that uses the textureless regime: the depth priors that make 3DGS work on blank surfaces also give an appearance-free signal for fixing SAM leaks.
Training alternates three steps. First, optimize Gaussians and offsets. Second, re-cluster votes with HDBSCAN as VoteSplat does, and update $c_k$ as a robust mean of the votes. Third, recompute $w_{k,v}$ and the corrected masks. The main risk is confirmation bias: early wrong clusters could veto correct masks. To limit this, weights are frozen at 1 during warm-up, and $w_{k,v}$ is bounded below so that no view is ever fully silenced.
Baselines
The naive reference point is VoteSplat run exactly as published, with unweighted SAM-centroid supervision, on the same 3DGS backbone and the same SAM masks. This shows how much error the unaudited cascade carries. VoteSplat is also the strongest existing method for this exact question, since it is the only cited approach that trains point-to-center offsets on Gaussians from 2D masks. It reports gains over LangSplat and OpenGaussian on LeRF, but there is no reported result found on Lacuna for VoteSplat's numeric mIoU or mAcc on LeRF, 3D-OVS, or GraspNet. We will therefore rerun it ourselves with its released settings, the same number of iterations, and the same HDBSCAN parameter search as our method. Beating it means a clear gain in 3D instance AP and center error on the truncation and textureless subsets described below, with no loss on the full set.
The second baseline is a non-learned lifting pipeline. It uses the same DN-Splatter-style geometry, back-projects SAM masks with rendered depth, associates them across views by 3D overlap, and takes the robust median of back-projected points as each object's center. This tests whether learned offsets matter at all.
The third baseline is geometry-only voting trained on sensor point clouds, VoteNet, which reports 58.6 mAP on ScanNetV2 (Deep Hough Voting), and MLCVNet, which reports 64.5% [email protected] on ScanNetV2 (MLCVNet). Both are fully supervised with 3D labels. We treat them as an upper reference obtained with 3D labels, not a competitor. We will convert our clusters to boxes for comparison on the same ScanNetV2 split.
For geometry, DN-Splatter reports an F-score of 0.9243 on MuSHRoom (DN-Splatter). Our method must not fall meaningfully below the backbone we rerun there.
The ablations remove each part of $w_{k,v}$ in turn: truncation only, agreement only, and no depth-edge mask splitting. The full model is also compared with a variant in which $w_{k,v}$ is computed once from the SAM masks alone, without 3D feedback. That variant separates "discard suspicious masks" from "audit masks against 3D".
Why not just…?
Why not just drop every SAM mask that touches the image border? This heuristic plausibly fixes much of the field-of-view failure, and it is a real threat, so it appears in our ablations as the "truncation only" variant computed from the 2D mask. It cannot address leaks on textureless boundaries or merges caused by occlusion, because those masks can sit fully inside the frame. It also discards partially visible views that still constrain the center along some directions. If border-dropping matches the full model on the textureless subset, we will report that the agreement check adds nothing and reduce the claim to truncation handling.
Why not skip learned offsets and just back-project masks with depth to get centers? Back-projected mask pixels lie on the visible shell of the object. As the LiDAR review stresses, the volumetric center is empty space, so a median of shell points is biased toward the camera side. This pipeline also inherits every SAM leak, because it has no mechanism to reject a bad mask. Our second baseline measures this directly through center error against ScanNet instance meshes.
Why not use a better 2D segmenter, or fuse image features into the Gaussians with cross-attention as SGFormer does? A better segmenter still produces a truncated mask for a truncated object, and truncation is a geometric bias, not a recognition error. Richer appearance features help least where our target failures live, on same-colored adjacent surfaces. SGFormer also relies on labeled LiDAR panoptic training, which our setting lacks. We test the first variant by swapping in higher-quality masks, using SAM at its largest setting with manual correction on a small subset. If the audit's gain shrinks toward zero with cleaner masks, that confirms the audit is fixing mask errors rather than doing something else.
Proposed Experiments
The main testbed is ScanNet and ScanNetV2, which provide posed RGB-D video, dense instance labels, and many textureless walls, cabinets, and doors. Replica and ScanNet++ add higher-fidelity geometry, and LeRF-OVS allows comparison in VoteSplat's own setting. We report 3D instance AP, the 3D center error of each matched instance against the ground-truth mesh centroid, and box [email protected] for comparison with the voting detectors. Geometry is tracked with Chamfer distance and F-score; MonoSDF reports a Chamfer distance of 0.042 on ScanNet, which serves as a surface reference.
The decisive checks are two stratified evaluations. The first is a controlled truncation sweep. We crop training images to progressively narrower fields of view, which raises the fraction of truncated (object, view) pairs while keeping everything else fixed. VoteSplat's center error should grow with truncation, and ours should stay flat. The second is a textureless-boundary subset: instances whose ground-truth boundary with a neighbor has low image contrast but a clear depth step. Here the depth-edge splitting should turn merged clusters into correct separate instances.
A third check tests the mechanism directly. For every (object, view) pair we compute the true bias of the SAM centroid from ground truth, and ask whether $w_{k,v}$ ranks pairs by that bias (area under the ROC curve). If the weights do not predict actual mask error, any gain in AP is coming from somewhere else, and the core claim fails.
References
- VoteSplat: Hough Voting Gaussian Splatting for 3D Scene Understanding
- View Invariant 3D Human Pose Estimation
- MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
- DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing
- Accurate Localization of 3D Objects from RGB-D Data Using Segmentation Hypotheses
- Bidirectional Propagation for Cross-Modal 3D Object Detection
- Deep Hough Voting for 3D Object Detection in Point Clouds
- MLCVNet: Multi-Level Context VoteNet for 3D Object Detection
- Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review
- SGFormer: Semantic-Geometry Fusion Transformer for Multi-modal 3D Panoptic Segmentation