Systems that track an unknown object's 3D position and orientation while building its 3D model from an RGB-D video stream, with no CAD model available, are now accurate and fast. Two examples are BundleSDF and 6DOPE-GS. They share one weak point. Each trusts a 2D segmentation mask, either drawn once in the first frame or propagated by a video segmenter such as SAM2, to decide which pixels belong to the object. When that mask clips a mug handle or includes the fingers holding the mug, the error is written into the 3D model and the tracker drifts. The authors of GSGTrack say so directly: a poor first mask leads to a failed track.
We propose to stop treating the mask as ground truth. Each 3D primitive in the object model carries a running belief over three labels: "object", "hand/manipulator", and "background". Two kinds of evidence update that belief over time. The first is what a 2D vision model says about the pixels the primitive projects to. The second is whether the primitive actually moves with the estimated rigid object pose. Pose optimization and model growth then weight every primitive by this belief. In plain terms, the object decides what belongs to it, because points that do not move rigidly with it are evicted, while semantic evidence covers the cases motion cannot decide, such as fingers in a stable grip.
Core Claim
Two failure modes hurt handheld tracking, and each defeats a single cue. A loose initial mask that includes fingers in a firm grip cannot be repaired by motion alone, because those fingers move rigidly with the object for long stretches. A 2D segmenter cannot reliably split object from hand at the contact boundary, which is exactly where Split4D reports Gaussians shared across bodies. Our claim is that accumulating both cues per primitive, in a probabilistic form that can be revised, makes model-free tracking robust to corrupted initial masks and hand contact. A single-cue robustness scheme cannot do this.
The motion cue turns the static-world violation that breaks reconstruction into a supervision signal. In a hand-object video, background is static in the camera frame, the object follows one rigid motion, and the hand follows a different, non-rigid one. Disagreement with the object's rigid motion is therefore evidence about ownership, not just noise.
Relation to Prior Work
The proposal builds on three groups of work that play different roles.
The first group motivates it. BundleSDF requires a first-frame mask and handles occluding fingers with a hand-designed reduced penalty on "uncertain free space". 6DOPE-GS initializes from SAM2 and reports lower accuracy than BundleSDF on HO3D under severe hand occlusion. GSGTrack, the RGB-only member, admits sensitivity to the first mask. None of the three ever revises which 3D primitives belong to the object. NVS-HO shows the same bottleneck for handheld novel view synthesis: Grounded SAM2 masks pass finger noise into reconstruction, and pose estimation is the primary failure mode.
The second group supplies the method. Continuous Semantic Splatting gives each 3D Gaussian ellipsoid a Dirichlet distribution over classes. A Dirichlet is a distribution over class probabilities whose parameters behave like accumulated evidence counts, so it can be updated in closed form. The method composites these distributions into per-pixel expected labels and variance maps. PCF-Lift shows that modeling lifted features probabilistically down-weights noisy boundaries, and reports 63.5% PQ-scene on ScanNet against 62.0% for a deterministic predecessor. Both explicitly assume a static scene with known camera poses. We reuse the Dirichlet-on-Gaussians machinery but move it into an object-centric frame whose pose is being estimated. We also add a motion-based evidence channel that neither paper has.
RobustSplat contributes a design constraint. It shows that 3D Gaussian Splatting densification, the step that spawns new primitives where rendering error is high, is triggered by transient objects and leaves floaters behind. It also shows that switching densification off makes results overly smooth. We therefore gate densification by ownership rather than disabling it.
The third group sits on the edge. Split4D separates moving bodies using per-Gaussian velocities and contrastive identity features without a video tracker. It targets non-rigid view synthesis rather than metric 6D pose, and it uses deterministic identity vectors, which blur at contact. Older single-image work such as Stochastic Congruent Sets, which weights registration points by soft classifier scores, and DenseFusion, whose per-pixel fusion degrades under bad masks, establish that soft weighting helps in static frames. They have no mechanism for evidence to accumulate or be retracted over a video.
Research Program
We build on the 6DOPE-GS pipeline. It is a Gaussian object field, a set of 3D Gaussians in the object's own coordinate frame. Poses come from feature matching followed by PnP-RANSAC (a standard solver that recovers pose from 2D–3D matches while rejecting outliers), and an online pose graph refines them. To this pipeline we add a Dirichlet parameter vector $\alpha_i \in \mathbb{R}^3_{>0}$ to every Gaussian $i$. One entry counts evidence for each label: object, hand, background.
At each keyframe $t$, the current object pose estimate $T_t$ is used to render the field. Each Gaussian then receives an update:
$$ \alpha_i \leftarrow \lambda\,\alpha_i + w_s\, s_{i,t} + w_m\, m_{i,t}, $$
The terms are as follows. $\lambda \in (0,1]$ is a forgetting factor that lets wrong early evidence decay. $s_{i,t}$ is the semantic evidence vector: the opacity-weighted average, over the pixels Gaussian $i$ covers, of soft scores from the SAM2 object mask logits, an off-the-shelf hand segmenter, and similarity between lifted DINOv2 features and an object feature prototype. $m_{i,t}$ is the motion evidence vector. It is computed from the depth residual of Gaussian $i$ under $T_t$ and under the identity (static) camera hypothesis. A small residual under $T_t$ supports "object", a small residual under the static hypothesis supports "background", and a large residual under both supports "hand". The weights $w_s$ and $w_m$ balance the two channels.
The belief feeds back into tracking in three ways. First, the photometric and depth terms of pose refinement are weighted per pixel by the alpha-composited object probability $\bar{p}_{\text{obj}}(u)$, using the probabilistic rasterization of Continuous Semantic Splatting:
$$ \mathcal{L}(T_t) = \sum_{u} \bar{p}_{\text{obj}}(u)\,\big\| I_t(u) - \hat{I}(u; T_t) \big\|_1 + \beta\, \bar{p}_{\text{obj}}(u)\,\big| D_t(u) - \hat{D}(u; T_t) \big|. $$
Here $u$ indexes pixels, $I_t$ and $D_t$ are the observed color and depth, $\hat{I}$ and $\hat{D}$ are the rendered color and depth under pose $T_t$, and $\beta$ balances the two terms.
Second, Gaussians whose expected hand or background probability stays above a threshold for $k$ keyframes are pruned. This lets a contaminated first mask be cleaned up after the fact.
Third, densification and spawning of new Gaussians are allowed only where both the rendered object probability is high and its Dirichlet variance is low. This follows RobustSplat's lesson to delay growth under uncertainty.
The research questions are whether this loop converges from badly corrupted masks, and which evidence channel does the work in which regime. We will also examine the circularity risk: a wrong pose produces wrong motion evidence, which in turn produces a wrong mask. The forgetting factor and keeping semantic evidence as an independent anchor are the intended safeguards, and we will test whether they are enough.
Baselines
The naive reference is 6DOPE-GS run unchanged with its SAM2 initialization. It is our starting codebase, so differences isolate our additions. The file reports only that it reconstructs at a "sub-centimeter level" on HO3D at 3.5 Hz. No reported ADD-S value was found on Lacuna for 6DOPE-GS, so we will rerun it ourselves.
The strongest existing RGB-D competitor is BundleSDF. Its hybrid SDF treatment of uncertain free space is the closest existing answer to hand occlusion. It reports 96.52% ADD-S and 0.57 cm Chamfer distance on HO3D. For YCBInEOAT, no reported numeric result was found on Lacuna, so we will rerun both systems. To keep the comparison fair, all methods receive identical initial masks: the clean SAM2 mask, and each corrupted variant described below. They run on the same GPU class, and each gets the same hyperparameter search budget on a held-out sequence subset.
Beating BundleSDF means two things. On clean masks, we must match its HO3D ADD-S within about a point. Under corrupted masks, we must keep ADD-S high while BundleSDF degrades. The headline claim is about robustness, not clean-mask accuracy.
For the RGB-only setting, GSGTrack is the baseline, with 64.60% ADD-S and 20.06 cm Chamfer distance on HO3D. We port the ownership mechanism into its silhouette loss.
Four ablations isolate the key ingredient. The first uses only semantic evidence ($w_m=0$). The second uses only motion evidence ($w_s=0$). The third replaces the Dirichlet with a deterministic per-Gaussian label overwritten each frame, which tests whether accumulation matters. The fourth applies a hard 0.5 threshold on ownership instead of soft weighting.
Why not just…?
Why not just re-run SAM2 every frame and subtract a hand segmenter's mask? This is the cheapest strong competitor. Its weakness is that it repairs nothing in 3D: fingers wrongly fused into the model in frame one stay there. NVS-HO reports that per-frame Grounded SAM2 masks still leave pose estimation as the dominant failure. We run this variant as a baseline, using the same SAM2 and hand segmenter fed into unmodified 6DOPE-GS. If it matches us under corrupted initial masks, the contribution reduces to 2D cleanup and we will say so.
Why not just use a robust loss or RANSAC to reject hand points geometrically? This is a real threat for moving fingers, and our motion-only ablation is essentially this approach. It should fail specifically during firm, stable grips, where finger points satisfy the object's rigid motion and robust estimators accept them as inliers. We will report results separately on grip-stable segments, identified from HO3D hand-pose annotations as frames with low relative hand-object motion. If motion-only matches the full model there too, the semantic channel is unnecessary and we will drop it.
Why not just rely on BundleSDF's hybrid free-space penalty? That penalty softens the loss in occluded regions but never reassigns geometry already absorbed from a contaminated first mask. The file notes this as an open vulnerability for BundleSDF. The mask-corruption sweep tests the point directly.
Why not use a feed-forward pose regressor such as VGGT or Dust3R? The evidence in the file is unfavorable. COLMAP beat VGGT on handheld captures in NVS-HO, and Dust3R reached 59.92 cm Chamfer distance on HO3D versus GSGTrack's 20.06 cm. We do not rerun these.
Proposed Experiments
The central experiment is a controlled mask-corruption sweep on HO3D and YCBInEOAT. It starts from a clean first-frame mask. We then dilate or erode it by fixed pixel radii, add the ground-truth hand region, which is rendered from the HO3D hand annotations, and remove the largest non-convex part of the object, such as a handle or protrusion. We plot ADD-S, ADD, and final-model Chamfer distance against initial-mask IoU for every method. The decisive result is a flatter curve for our method than for BundleSDF and 6DOPE-GS, with the motion-only and semantic-only ablations each failing in a different corruption type.
We also directly measure ownership quality. Rendered object-probability maps are scored against object masks obtained by rendering the ground-truth object mesh at the ground-truth pose. Accuracy is broken down by distance to the hand contact region and by grip-stable versus grip-changing segments.
A secondary RGB-only test applies the mechanism to GSGTrack on HO3D. Where NVS-HO data is available, we use its board sequences, which have ground-truth poses, to check whether ownership-weighted tracking reduces the pose bottleneck that benchmark identifies.
Runtime overhead against 6DOPE-GS's 3.5 Hz is reported. The Dirichlet update is closed-form, so we expect a small cost.
The proposal is falsified under either of two outcomes. One is that the per-frame-resegmentation baseline closes the gap under corruption. The other is that motion-only evidence matches the full model on grip-stable segments.
References
- BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects
- 6DOPE-GS: Online 6D Object Pose Estimation using Gaussian Splatting
- GSGTrack: Gaussian Splatting-Guided Object Pose Tracking from RGB Videos
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- NVS-HO: A Benchmark for Novel View Synthesis of Handheld Objects
- Modeling Uncertainty in 3D Gaussian Splatting through Continuous Semantic Splatting
- PCF-Lift: Panoptic Lifting by Probabilistic Contrastive Fusion
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- Robust 6D Object Pose Estimation with Stochastic Congruent Sets
- DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion