Most 3D reconstruction methods assume the scene holds still. When something moves, the optimizer treats the change as missing geometry and fills the air with leftover "floaters." Dynamic methods remove the assumption by learning a free-form motion field for every scene. With one camera and a narrow viewing path, that motion field can explain away almost anything, so shape and motion become entangled. Our proposal starts from the opposite observation. A rigid or articulated object being turned in someone's hand is static in its own coordinate frame. Once we know where that frame is at each moment, the object's motion stops being a nuisance and becomes free extra viewpoints of a single unchanging shape. The hard part is finding that frame for an object the system has never seen. Here a second line of work helps: category-level models describe a whole class, such as all mugs or all laptops, in one shared normalized frame and absorb the shape differences between instances.
We propose a 4D Gaussian splatting reconstruction of egocentric hand-object video. Gaussian splatting represents a scene as many small colored 3D blobs that are rendered by projection. In our version the object's blobs are tied to two things. The first is a time-invariant instance shape, drawn from a learned category shape space plus a small penalized residual. The second is a per-frame rigid or per-part articulated pose. Anything that cannot be explained this way is routed to a separate hand layer or to the static background, rather than spawning new object geometry. The claim we want to test is narrow. The category prior should matter beyond giving a good starting shape: by deciding what counts as "object," it prevents the motion field from absorbing shape errors. This should yield complete object geometry and stable poses where unconstrained dynamic Gaussians and model-free trackers fail.
Core Claim
In egocentric manipulation video, a dynamic reconstruction should constrain the object's time-varying part to lie in a category-defined family. That family is rigid motion, or articulation of category-known parts. The time-invariant part should lie near a category shape space. This factorization should beat two alternatives: free-form deformation fields, and per-frame category estimation followed by fusion. It should help both on occluded geometry, for example a handle hidden behind fingers in most frames, and on pose stability.
Concretely, the position of object Gaussian $i$ at time $t$ is
$$ x_i(t) = T_{p(i)}(t)\,\big(D(\mu_i; z) + r_i\big), $$
where $\mu_i$ is the Gaussian's location on the category template, $z$ is a single instance shape code shared across all frames, $D$ is a category deformation network that morphs the template into the instance, $r_i$ is a per-Gaussian residual penalized toward zero, $p(i)$ is the category part the Gaussian belongs to, and $T_{p}(t)$ is that part's rigid transform at time $t$. Nothing in this expression can vary with time except the transforms.
The second mechanism replaces the usual densification step, in which Gaussian splatting splits or clones blobs wherever rendering error is high. We gate this step for the object layer. A new object Gaussian is admitted only if its canonicalized location $T_{p}(t)^{-1}x$ is consistent across at least $k$ frames and lies within a distance budget of the deformed template. Otherwise the error is assigned to the hand layer or the background layer. This targets the floater mechanism directly: transient error no longer becomes permanent object geometry.
Positioning Against Existing Work
Three papers motivate the gap.
- HOI4D is a 4D egocentric RGB-D dataset covering 16 categories and 800 instances. It reports that the model-free tracker BundleTrack drops from about 86% on synthetic data to 19.3% on its real interaction video. This shows that ignoring category structure fails under hands and blur.
- Chord deforms category implicit templates to reconstruct hand-held objects and reaches Chamfer distance 7.69 on COMIC. It works on single frames, so nothing accumulates evidence over time.
- Deformable 3D Gaussians learns a free deformation field per scene and acknowledges overfitting and floaters under limited camera motion. Kinematics-GS adds velocity-aligned blob shapes for blurry video but is likewise per-instance with no shape prior.
The closest bridging work constrains the design.
- CaSPR maps point cloud sequences into a normalized space-time object frame, with a latent ODE and a normalizing flow. It requires dense ground-truth canonical coordinates for training and does not render appearance. We use canonical coordinates only through a prior trained on synthetic data, and fit each real video by self-supervised photometric and depth losses.
- CAPTRA shows that tracking small pose changes inside the previous frame's normalized frame beats per-frame estimation by 40.03% absolute at the 5°5cm threshold. It produces no geometry or rendering. We adopt its canonicalize-then-track idea to initialize $T_p(t)$, then refine it jointly with shape through rendering.
The category shape machinery comes from three sources.
- NOCS provides the shared normalized frame.
- DPDN deforms category priors in feature space with self-supervised consistency for transfer from synthetic to real data. We reuse its deformation network to produce $D$ and an initial $z$.
- CLA-NeRF discovers revolute joints at category level from static captures. We borrow the idea of category part labels and joint axes, but optimize continuous joint angles over a video.
The scene decomposition into background, hand and object follows two methods.
- Split4D separates moving and static Gaussians using motion and identity features, but treats moving Gaussians as unconstrained particles.
- RobustSplat delays densification so that static structure converges before transients are fit, and then discards the transients.
Our object layer is the part neither of them models: a moving entity that is kept, completed and posed.
Research Program
Category prior. We first train per-category priors on synthetic data. Each prior consists of a template point set, a DPDN-style deformation network, and part labels with joint axes for articulated categories. We then validate the priors on REAL275 single frames, so we know their standalone quality before they enter the 4D system.
Per-video fitting. For each HOI4D sequence we fit three Gaussian layers.
- Background layer. Static, with RobustSplat-style delayed growth.
- Hand layer. A free deformation field in the style of Deformable 3D Gaussians.
- Object layer. Parameterized as in the equation above.
Pose is initialized by running the category estimator on the first frames and propagating CAPTRA-style. We then optimize all layers jointly against RGB, depth and coarse 2D object and hand masks. The masks come from an off-the-shelf segmenter and are only used early, as soft guidance. The residuals $r_i$ carry an L2 penalty whose weight we sweep, because instances that sit far from the category mean will need them; the "novelty mug" problem is real.
Analysis. We will measure three things. First, how much of the final object geometry comes from $z$ and how much from $r_i$, as a function of how atypical the instance is. Second, whether gated densification actually lowers the count of Gaussians placed in hand-occupied space. Third, how the benefit of the prior scales with the fraction of the object surface that was ever observed. The last measure is the mechanism test. If the prior matters mainly through occlusion completion and the arbitration between shape and motion, its advantage should grow as observed coverage shrinks.
Baselines
Naive reference. The do-nothing reference is the per-frame category estimate itself: DPDN run independently on each masked frame, with shape taken from a single frame. It tells us what temporal aggregation adds. The strongest per-frame category result in the file is DPDN at 46.0% mAP for 5°2cm on REAL275 (supervised), from DPDN. On HOI4D we found no reported DPDN result on Lacuna, so we will run it ourselves using the same prior weights our method uses.
Strongest tracking competitor. This is BundleTrack (BundleTrack), a model-free 6D tracker with pose-graph memory. It is the strongest because it needs no category model and still reports 87.4% at 5°5cm on REAL275. The reported HOI4D figure is 19.3% tracking accuracy, from HOI4D. We will rerun it on our exact split and protocol rather than rely on that number, and give it the same initial masks. Beating it means a clearly higher 5°5cm rate on HOI4D under the same inputs.
Pose-only category tracker. CAPTRA is the closest category tracker. It tests whether joint reconstruction improves pose over tracking alone. It will be trained on the same synthetic data as our prior.
Strongest reconstruction competitors. These are Deformable 3D Gaussians and Kinematics-GS, each run per sequence on RGB-D with the same depth loss, iteration count and masks we use. We found no reported result on Lacuna for either method on HOI4D. Beating them means lower Chamfer distance of the object to the instance mesh, fewer Gaussians placed in hand or empty space, and higher held-out-frame PSNR within the object mask.
Ablations. Two ablations isolate the key ingredient. The first, prior-initialized free deformation, keeps our category initialization but then frees the object layer with an unconstrained deformation field. The second, generic-rigidity, replaces the category shape space with an as-rigid-as-possible regularizer and ordinary densification. Our method must beat both, or the category prior is only a warm start.
All methods get the same tuning budget, a fixed number of configurations searched on a validation split of HOI4D sequences.
Why not just…?
Why not just segment with a video segmenter, run DPDN or Chord on each frame, and fuse the results? This cascade is cheap and strong. Its weaknesses are already documented. CAPTRA shows that per-frame category estimation jitters and loses 40.03% absolute at 5°5cm against canonicalized tracking. Split4D shows that 2D mask errors under contact propagate into 3D. Per-frame shapes also disagree with each other, and averaging them gives no photometric check. We include this cascade as the naive baseline. If it matches us on Chamfer distance in high-coverage sequences, we will report that, and restrict our claim to low-coverage and occluded regimes.
Why not just initialize Deformable 3D Gaussians from the category template and let it run? This is the most serious threat. If the prior's only value is a good starting shape, this approach captures it for free. The prior-initialized free-deformation ablation tests exactly this. Our prediction is that the free field gradually trades shape for motion: the time-varying deformation absorbs errors that should have corrected the instance code. This would show up as drift over long sequences and as hallucinated geometry under hands. If the gap is negligible, the time-invariant factorization is not worth its constraints, and we would say so.
Why not just use generic physical regularizers like rigidity, with no category knowledge? Rigidity removes much of the shape-motion ambiguity for rigid objects. However, it cannot place geometry that was never observed, and it gives no part structure for laptops or scissors. The generic-rigidity ablation measures this. We expect parity on frequently seen surfaces and a gap that grows as observed coverage falls. That gap is the specific prediction tying the result to the category prior.
Why not just use a model-free tracker and fuse depth? BundleTrack's reported 19.3% on HOI4D argues against it as a pose source. Fusing depth can only reconstruct what was seen. We still run it as the tracking baseline rather than assume it fails.
Proposed Experiments
Main benchmark. We use HOI4D, which matches our target regime: real egocentric RGB-D, hand occlusion, intra-class variation, and both rigid and articulated categories. We split by instance, so test instances never appear in prior training. The metrics are:
- 5°5cm and 5°2cm pose accuracy, plus jitter measured as frame-to-frame pose variance;
- object Chamfer distance and F-score against the instance meshes released with the dataset;
- PSNR and LPIPS on held-out frames within the object mask;
- the share of object-layer Gaussians lying in hand or empty space, as a floater count.
For articulated categories we also report joint-angle error.
Supporting benchmarks. REAL275 validates the standalone prior. COMIC compares our object layer, run on short clips, against Chord's 7.69 Chamfer distance.
Decisive check. We bin HOI4D test sequences by the fraction of the object surface ever visible. Then we plot the Chamfer-distance gap between our method and the two key ablations, prior-initialized free deformation and generic rigidity. The proposal stands if the gap is near zero at high coverage and grows clearly as coverage drops, while pose accuracy stays at least at CAPTRA's level. It fails if the free-deformation ablation tracks our curve. A secondary check is gated densification switched off with everything else held fixed. If the floater count and hand-region contamination do not drop when the gate is on, the arbitration mechanism is not doing the work we claim.
References
- HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction
- CHORD: Category-level Hand-held Object Reconstruction via Shape Deformation
- Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction
- Kinematics-Driven Gaussian Shape Deformation for Blurry Monocular Dynamic Scenes
- CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations
- CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds
- Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation.
- Category-Level 6D Object Pose and Size Estimation Using Self-supervised Deep Prior Deformation Networks
- CLA-NeRF: Category-Level Articulated Neural Radiance Field
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models