Knowing What You Did Not See: Evidence-Labeled Object Proxies for Pose-Free, CAD-Free 6D Pose

A robot that has to pick up a new mug usually first builds a rough 3D model of it from a few photos, then estimates the mug's position and orientation (its "6D pose") by rendering that model and comparing it with the camera image. The rough model is built in one of two ways. The first is silhouette intersection, the "visual hull." It cannot see inside cavities, so it fills the mug's interior with solid material. The second is an image-to-3D generator. It invents geometry it never observed, and those inventions are often wrong around thin walls and concavities. Both methods return a model in which every surface looks equally trustworthy. The pose estimator therefore treats a made-up mug interior with the same confidence as a wall it actually photographed.

We propose a different object model: one that records, for every piece of space, whether it was carved (seen through to something behind it), verified (seen as a surface from several views that agree), or unobserved. Verification uses the agreement of 2D foundation-model features (DINOv2 descriptors, which change little under shading and highlights) across views, rather than raw color agreement. Camera poses come from a feed-forward pose-free geometry model, refined against the same feature agreement, so no Structure-from-Motion run or calibrated rig is needed. Pose scoring at test time then uses only carved and verified regions and stays neutral about unobserved ones. The claim is narrow and testable. On concave and hollow objects, keeping track of what was not seen should reduce pose failures that come from false geometry ("phantom volume"). It should do so better than either filling unseen space, as the visual hull does, or guessing it, as diffusion priors do.

Core Claim and How It Differs from Existing Work

The visual hull's blind spot is a basic limit of silhouettes, not a tuning problem. Silhouettes alone cannot tell a hollow cylinder from a solid one (Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency formalizes this ambiguity). Silhouette-trained generators fill the space under chair seats with phantom voxels (3D Shape Induction from 2D Views of Multiple Objects). Adding orthogonal views does not help (Tamura et al.). Cameras placed outside an object often simply cannot see into deep recesses (Zuo et al., Palanirajan et al.). These papers motivate the proposal. Some regions will stay unobserved no matter what, so the useful question is how a pose estimator should behave toward them.

The closest existing systems resolve unobserved geometry by committing to a guess.

  • Any6D builds a full mesh with InstantMesh from one RGB-D anchor and refines pose against it. It has no closed loop to correct a distorted proxy.
  • HIPPo comes closest in spirit. It replaces hallucinated Wonder3D surfaces with Poisson-reconstructed measurements as new views arrive. However, it needs metric depth and treats the current mesh as uniformly valid during tracking.
  • FoundationPose fits a neural signed distance function from posed reference views. Fast fits of this kind tend to smooth over concave openings.

None of the three exposes per-region evidence to the pose scorer, and FoundationPose requires reference poses.

Two older methods supply the carving mechanism.

- Stochastic Refinement of the Visual Hull shrinks a visual hull inward using photometric consistency while pinning silhouette contact points. It recovers bowl basins and armchair seats, but it needs calibrated cameras, keeps the mesh topology fixed, and assumes matte surfaces.

  • Differentiable ray consistency carves concavities once depth or color enters the loss, again with known poses.

Our verification step replaces their raw-color agreement with foundation-feature agreement. It runs on an occupancy representation that can change topology, under estimated rather than calibrated poses.

On the pose-free side, Learning Compact 3D Gaussians via Feed-Forward Point Fusion (SPLATT3RFUSION) already shows two things: a geometry transformer of the VGGT family can recover structure from unposed images, and learned-feature similarity is a reliable signal for merging 3D points. It is scene-level, object-agnostic, and has no notion of unobserved space. We reuse its premise that feature similarity is a trustworthy 3D consistency signal, and apply it to object-level verification and carving.

Research Program

Onboarding. Onboarding takes 8 to 30 unposed RGB images of a novel object, as in the BOP model-free onboarding videos. The steps are:

  1. Segment the object with SAM.
  2. Get initial camera poses and per-view depth from a feed-forward pose-free geometry model.
  3. Extract dense DINOv2 features in every view.
  4. Discretize the region inside the initial visual hull into a sparse octree of cells.

Evidence states. For each cell $x$, let $V(x)$ be the set of views in which $x$ is the first occupied point along the camera ray under the current occupancy estimate. We define a feature-agreement score

$$c(x) = \frac{1}{|V(x)|(|V(x)|-1)} \sum_{i \neq j \in V(x)} \cos\big(f_i(\pi_i(x)),\, f_j(\pi_j(x))\big),$$

where $f_i$ is the feature map of view $i$ and $\pi_i$ projects a 3D point into view $i$ under the current pose estimate. Each cell then gets one of three states:

  • Verified surface: $|V(x)| \ge 2$ and $c(x)$ exceeds a threshold.
  • Carved: some ray passes through $x$ and lands on a verified surface behind it, or on background.
  • Unobserved: everything else. This covers a mug interior seen only at grazing angles, and the underside of an object resting on a table.

The procedure alternates two steps until it converges: carve cells whose visible surface fails feature agreement, then re-estimate camera poses by maximizing $\sum_x c(x)$ over the verified cells. The second step is a feature-metric bundle adjustment restricted to the object.

This uses feature agreement only as a score. We do not train a fused RGB-geometry network, so the modality-collapse problem in Countering Multi-modal Representation Collapse through Rank-targeted Fusion does not arise.

Test-time pose estimation. Given a query image, metric depth from the query RGB-D image (as in BOP), and pose hypotheses, we render the proxy and score each hypothesis $T$ as

$$S(T) = \sum_{p \in \Omega_{\text{ver}}(T)} r_p(T) \;+\; \lambda \, |\Omega_{\text{unobs}}(T)|.$$

Here $\Omega_{\text{ver}}(T)$ is the set of pixels whose rendered first surface is verified or carved-through-to-verified, and $r_p$ is a robust depth-plus-feature residual at pixel $p$. $\Omega_{\text{unobs}}(T)$ is the set of pixels whose ray first hits unobserved space, and $\lambda$ is a small penalty that stops the optimizer from hiding the object in its own blind spots.

We plug this score into FoundationPose's hypothesis generation and refinement in place of its scoring network's input mask. Everything else stays the same, so the comparison isolates the evidence labels.

Geometry measurements. On objects with ground-truth meshes, we also measure phantom volume: the fraction of predicted occupied volume lying outside the ground-truth solid. This tests whether the carving step works independently of pose.

Baselines

The naive reference point is the visual hull itself. We build it from SAM masks and the same feed-forward poses, use it as the proxy, and score every rendered pixel. It is what the carving step must improve on. If our proxy does not clearly reduce phantom volume relative to it on bowls, mugs and pitchers, the method has failed at its first stage.

The strongest reconstruct-then-track competitor is FoundationPose in its model-free mode. It gets the same onboarding images, with reference poses from COLMAP where COLMAP succeeds and from ground truth as an upper bound. FoundationPose reports 99.9% ADD-0.1d on LINEMOD in the model-free setting, and HIPPo reports FoundationPose at 79.71 ADD-S on YCB-Video.

The strongest prior-based competitor is HIPPo, which reports ADD-S of 97.00 on YCB-Video from a single reference image. Any6D reports 98.7% mean ADD-S on HO3D and 45.6% strict ADD on YCBInEOTA.

Because these aggregate numbers are close to saturation, "beating" these methods is defined on the concave-object subset: YCB bowl, mug and pitcher; LINEMOD bowl and cup; and the REAL275 bowl and mug instances, evaluated instance-wise without category training. We did not find per-object numbers for these objects on Lacuna, so we will rerun all baselines with released code, the same onboarding images, the same query depth, and the same GPU budget. The target is a statistically significant drop in failure rate at strict ADD on that subset, with no regression on convex objects.

Three ablations isolate the key ingredients:

  • Two-state labels: treat unobserved cells as occupied (visual-hull behavior), and separately as empty.
  • Color instead of features: replace DINOv2 agreement with raw RGB agreement, as in Isidoro-style photometric carving.
  • Fixed poses: keep the feed-forward poses fixed rather than refining them.

Why not just…?

Why not just fuse the feed-forward model's depth maps into a mesh and hand it to FoundationPose? This is the most serious threat. Predicted depth from many views already carves any concavity that is actually visible, without silhouettes. Where it may fall short is unobserved regions. Depth fusion either leaves holes, which the pose scorer reads as empty, or interpolates over them, which produces the same false confidence we are trying to remove. The proposal includes this exact pipeline (feed-forward depth plus truncated signed distance fusion plus FoundationPose) as a baseline. If it matches our method on the concave subset, then the evidence labels add nothing and we will report that.

Why not just use an image-to-3D generator, as Any6D and HIPPo do? Their own descriptions name distorted thin and concave geometry as the failure mode, and HIPPo's need to overwrite hallucinated surfaces with measured ones shows that the authors do not trust the prior. The comparison above measures directly whether ignoring unseen regions beats guessing them. One hybrid is worth testing as well: use the diffusion prior only inside unobserved cells, with the down-weighting applied there. If that hybrid wins, the contribution becomes the labeling scheme rather than abstaining from priors.

Why not just run COLMAP for onboarding poses and carve with color agreement? COLMAP is fragile on low-texture and sparse captures. OnePose++ shows keypoint-based pipelines dropping from 57.7% to 35.7% success on OnePose-LowTexture. Color-based carving also assumes matte surfaces, which glossy mugs violate (GlossyGS). The color-versus-feature ablation and the COLMAP-pose variant of FoundationPose test both points. We will also report how often COLMAP fails to register on each dataset.

Proposed Experiments and Decisive Checks

Geometry. On YCB objects with ground-truth meshes, and on the calibrated armchair and bowl images from Isidoro et al., we report Chamfer distance and phantom volume for four proxies: visual hull, depth fusion, generative proxy, and ours. The armchair is the reference point: Isidoro et al. report its average pixel error falling from 0.097 to 0.040 with calibrated poses. The question is whether our method reaches comparable geometry from unposed input.

Pose. Pose evaluation covers YCB-Video, LINEMOD, REAL275 instances, HO3D, and the BOP-H3 model-free onboarding track (HOT3D, HOPEv2, HANDAL), which has no per-dataset numbers on Lacuna. We stratify results by an object-concavity measure, defined as the visual-hull-to-ground-truth volume ratio, and report ADD, ADD-S and BOP average recall per stratum.

Decisive checks. Two results decide the project. The first is the correlation, across objects, between phantom-volume reduction and pose-error reduction. If carving improves geometry but not pose, the motivation collapses. The second is the comparison with depth fusion on the concave stratum. A further diagnostic tracks how pose accuracy changes as onboarding views are removed from above the object. Unobserved space grows as those views disappear, so this is where our method should degrade more gracefully than every baseline. If it does not, the core claim is wrong.