A robot often has to find the position and orientation of an object it has never seen before. This is the object's 6D pose: three numbers for where it is and three for how it is rotated. Often the robot has no 3D CAD model of the object, only one depth-camera snapshot of it. Current methods that work from a single snapshot split into two camps. The fast camp matches image features between the snapshot and the live view, and it breaks when the live camera sees a side the snapshot never showed. The accurate camp first runs an image-to-3D diffusion model (a generator that turns a picture into a full 3D shape) and then tracks the resulting mesh. It handles unseen sides, but it pays seconds of latency and inherits the generator's shape errors.
We propose a middle path. The reference snapshot is turned into a sparse voxel grid: only the occupied cells of a 3D grid are stored, each holding a learned feature vector. A decoder in that grid then grows new cells into the unobserved back side and interior of the object, borrowing the "add cells, then prune unlikely ones" machinery that 3D detectors use to fill hollow object centers. Each point of the live depth scan is then matched against this completed feature grid in a single forward pass, and the pose is solved in closed form. The bet is that pose estimation does not need a renderable mesh. It needs features in the right places on the hidden side, and growing those directly in 3D should be both fast and good enough.
Core Claim
Coordinate-expanding sparse decoders are a generator that runs in milliseconds and works directly in 3D. Trained on synthetic objects to complete reference features rather than surfaces, such a decoder should close most of the accuracy gap between feed-forward one-reference pose estimators and diffusion-based pipelines on low-overlap query views. It should do this at roughly feed-forward latency. The claim is falsifiable on two axes. On views where the query shares little visible surface with the reference, the method must clearly beat 2D-matching methods. Against Any6D or HIPPo, it must approach their accuracy while being one to two orders of magnitude faster. If either fails, the idea fails.
Positioning in Prior Work
Several papers motivate the gap. POPE chains Segment Anything, DINOv2 retrieval and LoFTR correspondences into a solver over 2D image-to-image matches. One2Any predicts, for every query pixel, its coordinates in the reference camera's frame (Reference Object Coordinates), using a 2D U-Net conditioned on an encoding of the reference. It reaches 54.9 Average Recall (AR) on REAL275 at 90 ms. Both methods compare 2D evidence, and the research file records that such matching fails when query and reference views are disjoint. We keep One2Any's output target, coordinates in the reference frame, but predict it from a 3D reference representation that has been extended to the unseen side.
Two papers fill that unseen side with external generators. Any6D builds a proxy mesh with InstantMesh and refines pose by multi-hypothesis render-and-compare. HIPPo combines Wonder3D and MASt3R, then tracks with FoundationPose. Both are accurate: Any6D reports 98.7% ADD-S on HO3D, and HIPPo reports 97.00 ADD-S on YCB-Video. HIPPo's initial mesh takes about 8 seconds, and Any6D degrades on thin and non-convex shapes. We replace the external meshing step with completion done inside the network. Unlike these methods, we never produce a watertight mesh.
The method itself comes from sparse convolution. Submanifold sparse convolutions keep the set of active cells fixed from layer to layer, which lets deep encoders stay cheap. GSDN adds transposed sparse convolutions with learned pruning to create cells in the empty centers of scanned objects; this lets it process a 78-million-point building in 5 GB. SparseAlign expands coordinates during downsampling for the same hollow-center problem in LiDAR. Both GSDN and SparseAlign stop at bounding boxes for closed-set classes. We repurpose their expansion to complete a single novel object conditioned on a reference.
Two papers constrain the design. Sparse Steerable Convolutions is the closest sparse-voxel pose work: 43.4% at 5°5cm on REAL275 and 404 FPS. It needs CAD models or known categories, and its fixed-support layers cannot reason about occluded space. Gen6D shows that a 3D feature volume built from reference images can refine pose without CAD. Its volume is dense, which caps resolution, and it is RGB-only, which makes it sensitive to depth errors. We use depth for unprojection and sparse storage for resolution.
Research Program
The reference RGB-D shot, segmented by a prompt mask, is unprojected into a sparse tensor. Each observed surface point becomes an occupied voxel, at roughly 5 mm resolution normalized by object size. Each voxel carries frozen DINOv2 patch features, color, and a continuous within-voxel offset to limit quantization error. A submanifold sparse U-Net encodes this shell. A generative sparse decoder then upsamples with transposed sparse convolutions. At each scale, a pruning head predicts whether each candidate cell lies on the object's true surface. Surviving cells receive features and a predicted reference-frame coordinate.
Training uses synthetic RGB-D renders of Objaverse and Google Scanned Objects, the asset pool One2Any used. CAD meshes appear only at training time, on objects disjoint from the test sets, to supervise occupancy and features on the hidden side. The feature target for a hidden voxel is the encoder feature that the same voxel receives when that side is actually observed in another render. Completion is therefore supervised in feature space, not in appearance.
The query depth scan is voxelized the same way and encoded with a shared submanifold encoder. Cross-attention restricted to nearby candidate cells maps each query voxel to a coordinate in the reference frame, with a per-voxel confidence. Pose and metric scale come from Kabsch-Umeyama with RANSAC over confident voxels. An optional single refinement step on raw points corrects residual quantization error.
Two failure modes from the sparse-convolution literature are designed against. The first is runaway expansion against over-pruning. We cap active cells per scale and anneal the pruning threshold, and we report cell counts. The second is limited non-local context, which matters most for matching the front of a query to the back of the reference. We test it by ablating global attention against purely local attention.
Baselines
The naive reference point is direct partial-to-partial geometric registration between the reference and query point clouds with no learned completion. Any6D reports such a learned-descriptor baseline, Gedi, at 71.9% ADD-S on HO3D and 7.7% ADD on YCBInEOTA. We will rerun this registration on our exact splits and masks.
The strongest feed-forward competitor is One2Any. Its reported number is 54.9 AR on REAL275 at 0.09 s. It is the most direct threat because it answers the same question in the same output space without completion. We will run it with its released weights and its training data, and we will also retrain our model on the same synthetic set if that is feasible, so that data is not the confound.
The strongest accuracy competitors are HIPPo and Any6D. Their reported numbers are 97.00 ADD-S on YCB-Video for HIPPo, 98.7% ADD-S on HO3D for Any6D, and 45.6% ADD on YCBInEOTA for Any6D. We run them with the same reference frame and masks and report wall-clock time on the same GPU.
SAM-6D (70.4 AR on YCB-V) uses CAD models. We report it only as a ceiling for what full geometry buys, not as a peer.
Beating One2Any means a statistically clear gain in AR or ADD-S on low-overlap views at under 3x its latency. Matching the diffusion pipelines means coming within a few points of their ADD-S at under one second end to end.
Two ablations isolate the key ingredient. The first removes expansion, so the reference is a submanifold-only shell and nothing is grown. The second replaces sparse storage with a dense 3D CNN volume, Gen6D-style, at matched GPU memory, which forces a coarser grid.
Why not just…?
Why not just match 2D foundation features, as POPE and One2Any do? On views that overlap the reference, this may be as good as our method, and we expect it to be. The claim lives entirely in the low-overlap regime. We will bin every test pair by the fraction of the query's visible surface that the reference also observed, computed from ground-truth meshes and used only for evaluation. A flat curve for One2Any across bins would refute our motivation.
Why not just generate a mesh with InstantMesh or Wonder3D and track it with FoundationPose? This is a real threat. HIPPo's 97.00 ADD-S on YCB-Video suggests it may simply be more accurate. Our counter is not accuracy everywhere but the accuracy-latency trade-off. We will report full curves, including a latency-matched variant that runs fewer diffusion steps or render-and-compare hypotheses. We will also report a thin and non-convex object subset, where Any6D is documented to struggle. If diffusion pipelines dominate at equal time budgets, the proposal reduces to a speed engineering result, and we will say so.
Why not just complete the point cloud geometrically and then register it? This tests whether completing features matters beyond completing shape. An ablation trains the same expanding decoder to output occupancy only, then registers with descriptors computed on the completed geometry. If this matches the full model, the contribution is shape completion, and the feature supervision is unnecessary.
Why not just use a dense 3D volume like Gen6D? The dense ablation at matched memory answers this directly. We expect it to lose at fine rotation thresholds, because it must use a coarser grid.
Proposed Experiments
Training is on Objaverse and GSO renders only. Evaluation is zero-shot on four sets. REAL275 is used for comparison with One2Any, with AR and 5°5cm. YCB-Video is used for comparison with HIPPo, with ADD-S and AR. HO3D and YCBInEOTA are used for comparison with Any6D, with ADD-S, ADD and scale error. LM-O covers occlusion; the file has no reported Lacuna number for the model-free methods there, so all LM-O figures will be our own reruns. Every metric is additionally reported per overlap bin, together with latency and peak memory on one GPU.
The decisive check is the lowest overlap bin. There the method must beat One2Any by a clear margin and land near Any6D or HIPPo at a small fraction of their time. The no-expansion ablation must collapse toward One2Any in that bin. A secondary check correlates per-object completion quality, measured as feature cosine error on held-out sides, with pose error. A weak correlation would mean the gains come from elsewhere.
References
- POPE: 6-DoF Promptable Pose Estimation of Any Object, in Any Scene, with One Reference
- One2Any: One-Reference 6D Pose Estimation for Any Object
- Any6D: Model-free 6D Pose Estimation of Novel Objects
- HIPPo: Harnessing Image-to-3D Priors for Model-Free Zero-Shot 6D Pose Estimation
- 3D Semantic Segmentation With Submanifold Sparse Convolutional Networks
- Generative Sparse Detection Networks for 3D Single-Shot Object Detection
- SparseAlign: a Fully Sparse Framework for Cooperative Object Detection
- Sparse Steerable Convolutions: An Efficient Learning of SE(3)-Equivariant Features for Estimation and Tracking of Object Poses in 3D Space
- Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Images