When a person turns an object in their hand in front of a camera, the camera never sees the parts under the fingers. Current reconstruction systems either leave those parts hollow or fill them in with a generative model. A generative model here is a pretrained image or multi-view diffusion model, meaning a generator that learns by gradually turning random noise into samples. The model has no idea that a hand occupies that space, so the invented surface often passes straight through the fingers. We propose to reconstruct such scenes as a variable number of 3D "slots": one slot per physical entity, meaning hand, held object, other objects, and background. Each slot is a cluster of 3D Gaussians (small soft blobs used as rendering primitives) with its own motion. Two rules govern the slots. First, geometry the camera never saw may be invented by the generative prior, but it may not occupy space already owned by another slot. Second, persistent violations of that exclusivity count as evidence that the slot decomposition itself is wrong, and they drive slots to be split or merged.
In short, physical non-overlap does two jobs that are usually handled separately. It constrains what a diffusion prior is allowed to hallucinate in occluded contact regions. It also serves as the signal for deciding how many objects are in the scene. The regime is where these problems collide hardest: handheld and hand-object manipulation video, where the NVS-HO benchmark shows that current Gaussian Splatting and NeRF pipelines fail badly under hand occlusion.
Core Claim
The claim is that the occupancy of neighbouring entities is the missing grounding signal for generative completion in contact-rich dynamic scenes. The claim also covers what makes that signal usable: exclusion can only be enforced if the entity decomposition is correct, and exclusion violations are themselves a cheap diagnostic of a wrong decomposition. Adding each ingredient separately does not deliver this coupling.
The closest existing work each covers one side. AnimatableDreamer uses canonical score distillation to complete a single non-rigid entity from monocular video. Score distillation means pushing rendered views toward what a diffusion model finds likely. The method has no notion of other bodies it might collide with. 4DGS-WAM and Split4D decompose dynamic scenes into moving Gaussian entities, but they model no interaction between them. 4DGS-WAM states that it cannot hallucinate missing geometry and does not account for collisions. Split4D reports that Gaussians become shared across entities in tight contact zones.
Contact-physics methods do enforce exclusion. CPF uses attractive and repulsive virtual springs, and ContactOpt uses a differentiable contact model that tolerates 1–2 mm of soft-tissue penetration. Both require a known CAD mesh of the object, which is exactly what is missing when the object is only partly seen. We do not propose a new generative model or a new collision primitive. We propose the coupling between them and a slot-count rule derived from it.
Research Program
Representation. The scene is a pool of up to $K_{\max}$ slots. Each slot holds a set of Gaussians in a canonical frame, plus a per-frame rigid transform and a small non-rigid residual, following the actor parameterisation of 4DGS-WAM. The hand slot is initialised from a parametric hand model with anatomically constrained joints, in the spirit of A-MANO from CPF. Background Gaussians use delayed densification as in RobustSplat. That paper shows moving objects otherwise cause floating Gaussians to accumulate. Each slot carries a keep/drop gate, trained with the Gumbel-Softmax relaxation and sparsity penalty of AdaSlot. Dropped slots are zeroed in rendering so they cannot leak signal.
Exclusion. We use the fact, exploited in real-time two-hand tracking from depth, that the overlap of two Gaussians has a closed form. For Gaussians $i$ and $j$ in different slots at frame $t$, with means $\mu_i,\mu_j$, covariances $\Sigma_i,\Sigma_j$ and opacities $\alpha_i,\alpha_j$, the exclusion energy is
$$E_{\text{excl}}(t)=\sum_{i\in a,\,j\in b,\,a\neq b} w_{ij}\,\alpha_i\alpha_j\,\mathcal{N}(\mu_i;\mu_j,\Sigma_i+\Sigma_j+\delta^2 I),$$
where $\delta$ is a contact tolerance of roughly 1–2 mm, taken from ContactOpt's soft-tissue allowance. The weight $w_{ij}$ is large when either Gaussian lies in a region that was never observed and small when both are photometrically supervised. The generative prior is therefore only restricted where it is doing the inventing. A separate attraction term, a softened version of CPF's springs, pulls hallucinated object surface toward hand regions that were predicted to be in contact. This counteracts the floating failure that pure repulsion causes.
Generative completion. The object slot gets score distillation from a view-conditioned diffusion model in its own canonical frame, as in AnimatableDreamer's canonical anchoring. This avoids the per-frame drift that plain score distillation suffers. The distillation gradient and $E_{\text{excl}}$ act on the same Gaussians in the same optimisation step. Hallucinated mass that would enter the hand is therefore relocated, not deleted afterwards.
Exclusion-driven slot count. Every $T$ iterations we compute two statistics per slot pair: accumulated overlap, and the difference between their estimated rigid motions. Two slots that overlap persistently while moving identically are a fragmented single object, the "phantom objects" of Weis et al., so they are merged. A single slot whose Gaussians fit two distinct rigid motions far better than one is over-grouped, so it is split into a free slot from the pool. The AdaSlot gate then decides whether each slot survives. This replaces AdaSlot's purely photometric criterion with a 3D physical one. AdaSlot's own paper names part-whole and dense-object ambiguity as open problems, and that is exactly where photometric evidence is weak.
Baselines
The naive reference point is vanilla Gaussian Splatting (Splatfacto) run on the handheld sequences with object masks. NVS-HO reports it outperforming Nerfacto but still performing poorly in absolute terms. No reported median PSNR/SSIM/LPIPS was found on Lacuna for NVS-HO, so we will re-run it ourselves under the benchmark's protocol.
The strongest decomposition competitor is Split4D. It separates dynamic entities natively, without 2D video tracking, and reports 0.893 mIoU on Multi-Human against 0.592 for SA4D. It is the most credible method for getting entity boundaries right in 4D, which our slot mechanism must match before completion even matters.
The strongest completion competitor is a per-object adaptation of AnimatableDreamer's canonical score distillation. It reports 3.65 cm Chamfer distance on Cat-Coco against 10.7 cm for BANMo. We run it on the object slot with the same diffusion model and the same optimisation schedule as ours, but without exclusion or attraction terms.
For hand-object plausibility on HO-3D, the reference is ContactOpt. It reports MPJPE improving from 57.7 mm to 48.1 mm, but it is given the ground-truth object mesh. We report it as an oracle-object upper reference, not as a like-for-like competitor.
All methods share the same input frames, camera estimates, diffusion checkpoint, iteration count and per-scene wall-clock budget. Loss weights are tuned on a held-out split of five NVS-HO objects with an equal number of trials per method.
Three ablations isolate the key ingredients. The first applies exclusion uniformly, with $w_{ij}$ constant, to test whether restricting it to hallucinated regions matters. The second removes attraction, to measure floating. The third uses a fixed large $K$, with no gating or merge/split, to test the slot-count mechanism.
Beating the baselines means lower penetration depth and intersection volume against the hand than completion-only score distillation, with no loss in held-out-view PSNR, and object Chamfer distance on HO-3D at least matching it.
Why not just…?
Why not just segment with SAM2, reconstruct each object separately, and complete it with a diffusion prior? This is the most natural pipeline and it is our completion baseline with external masks. Split4D documents that tracker masks jitter and fail during fast motion and interaction, which drops SA4D to 0.592 mIoU. NVS-HO found Grounded SAM2 plus COLMAP/VGGT did not yield usable models of handheld objects. Even with perfect masks, independent completion has no information about where the hand is. We run this pipeline explicitly, so its penetration numbers will show whether that matters.
Why not just complete first and then delete any Gaussians that fall inside the hand? This is a real threat, and it is the ablation we care most about. Carving is cheap and fixes the penetration metric by construction. The counterargument is that carving removes mass without replacing it. The fingertip region becomes a dent, whereas joint optimisation lets the diffusion gradient re-place the surface just outside the contact tolerance. The decisive measure is therefore object Chamfer distance against HO-3D ground-truth object meshes in the contact region, not penetration. If carving matches us there, the completion half of our claim fails. The contribution would then reduce to the exclusion-driven slot count, which carving cannot provide, and we would report it that way.
Why not just use a large fixed number of slots? AdaSlot shows fixed large capacity fragments objects: a 33-slot model reaches 20.83 ARI on COCO against 39.00 with adaptive slots. 3D-DLP operates with a rigid 24–40 particle budget and imposes no non-overlap between particles. Fragmentation matters more in our setting than in 2D. A fragmented object applies exclusion against itself and tears its own hallucinated interior apart, so the fixed-$K$ ablation should show a larger gap than in AdaSlot.
Why not just learn a feed-forward network that corrects penetrations? ContactOpt compared exactly this, a learned RefineNet, against explicit contact optimisation and found the learned refiner clearly worse. Such a network would also need training data of completed occluded objects, which does not exist for novel objects. We do not re-run this comparison.
Proposed Experiments
The primary benchmark is NVS-HO. We train on the handheld sequences and evaluate PSNR, SSIM and LPIPS on the paired ChArUco board sequences, which show the object unoccluded. This directly measures whether hallucinated hidden sides are correct.
The second benchmark is HO-3D using the v3 annotations from HO-3D_v3. Its ground-truth hand and object meshes let us measure three things on reconstructed, not given, objects. The first is object Chamfer distance, overall and restricted to within 1 cm of the hand. The second is penetration depth and solid intersection volume, following CPF. The third is contact-region floating distance. We also report the Bullet gravity-hold test from CPF on the reconstructed object mesh.
For slot counting, we composite HO-3D and NVS-HO clips to contain zero, one or two extra objects, sometimes a second hand, and objects that are handed over or stacked. We report counting accuracy and per-entity mIoU against Split4D. We also run Multi-Human to check that the merge/split rule does not hurt Split4D's setting, where contact is between people.
Two checks decide the claim. The first is the contact-region Chamfer comparison against complete-then-carve. It must show a gap in our favour, or the joint-optimisation argument is dropped. The second is a stacking and hand-over subset, where two touching objects share a rigid motion for part of the clip. There the exclusion-driven merge rule must avoid permanently merging them, and it must beat AdaSlot-style photometric gating on counting accuracy. If both checks pass, the note's central claim holds: occupancy by neighbouring entities is what makes generative completion and entity counting trustworthy in contact-rich video.
References
- NVS-HO: A Benchmark for Novel View Synthesis of Handheld Objects
- AnimatableDreamer: Text-Guided Non-rigid 3D Model Generation and Reconstruction with Canonical Score Distillation
- 4DGS-WAM: Bridging Past and Future with an Object-Centric World Action Model based on 4D Gaussian Splatting
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- CPF: Learning a Contact Potential Field to Model the Hand-object Interaction
- ContactOpt: Optimizing Contact to Improve Grasps
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- Adaptive Slot Attention: Object Discovery with Dynamic Slot Number
- Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera
- Benchmarking Unsupervised Object Representations for Video Sequences
- 3D-DLP: Self-supervised 3D Object-centric Scene Representation Learning
- HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Dataset