Can a Depth-Only Pose Estimator Learn What DINOv2 Sees?

The most accurate systems for estimating the 3D position, orientation and size of an unfamiliar object, such as a mug the robot has never seen, from a colour-plus-depth camera now run a large pretrained image network alongside a point-cloud network at test time. The image network is DINOv2, a self-supervised vision transformer whose features encode object parts and semantics. Its features help the system tell a mug's handle side from its back and resolve near-symmetric shapes. The cost is speed, and these pipelines are slow. This proposal asks whether that image knowledge can be moved into the depth stream during training. The idea is to train a small point-cloud network to predict, at every 3D point, the DINOv2 feature it would have received. At test time the image backbone is dropped entirely. The student runs from depth alone, and is told by its training which instances needed the image the most.

The claim is deliberately falsifiable. Most of the accuracy that foundation-model image features add to category-level pose comes from cues that are inferable from local and global geometry, and so can be distilled into a geometry encoder. The part that cannot be recovered is concentrated where depth itself fails: transparent, reflective and thin objects. If this is right, the field's speed–accuracy tension largely disappears for ordinary objects. It would then reduce to a narrow, measurable residual that tells us exactly when a camera image is worth its cost. If it is wrong, the experiments show which cues are truly appearance-only.

Background and Gap

Category-level pose estimation predicts rotation, translation and per-axis size for unseen instances of a known category. The usual route since NOCS maps observed pixels to a shared canonical coordinate cube and aligns them to depth. Two recent lines of work pull in opposite directions. The first adds richer image features and heavier fusion. 6D-ViT runs parallel image and point transformers joined by cross-attention. CatFormer deforms a category template over five recurrent steps and runs at about 8 FPS. GCE-Pose completes the shape and injects DINOv2 features, reporting a 16% gain on HouseCat6D 5°2cm while its authors call real-time use open. DiffusionNOCS finds DINOv2 features important for symmetry and trains with random modality dropout. These papers motivate the proposal: they show image semantics matter. The second line strips cost. Query6DoF replaces dense shape templates with a few learnable query vectors that attend to point tokens, and reaches 49.0% 5°2cm on REAL275 at 34.9 FPS using points only. This paper supplies the student architecture.

Distillation work connects the two lines only indirectly, and each paper differs from the proposal in a specific way. ELiTe distills SAM image features into a 1.9M-parameter LiDAR segmenter that runs without cameras. It supplies the patch-to-point transfer technique, but it works on scene-level labels, not metric object pose. 3D-Augmented Contrastive Knowledge Distillation goes the other way, from 3D into an RGB-only student. It inherits monocular scale ambiguity and is evaluated on orientation, not full 9D pose. CleanPose distills a 3D foundation model, ULIP-2, into a pose network for debiasing rather than to remove an expensive input stream. Modality-specific Distillation shows that distilling a multimodal teacher as a single block leaves a gap. Weighting per-modality signals per instance closes part of it: 68.30% vs 66.53% AUC on Hateful-Memes. That evidence comes from classification only, but it constrains the design here, because plain output distillation is unlikely to be enough. No paper in the file distills foundation-model image features into a depth-only category-level pose estimator, or measures where such a transfer fails.

Research Program

A teacher and a student share one architecture except for the image stream. Both are Query6DoF-style networks: learnable queries attend to point tokens and regress rotation, translation and size.

The teacher additionally receives frozen DINOv2 patch features. These are sampled at each point's image projection and fused into the point tokens. The teacher is trained with random image dropout, following DiffusionNOCS, so the same network can produce both a fused prediction and an image-free prediction for every training instance.

The student has no image input. It carries a small head that predicts a DINOv2-like feature at each point. Its training loss combines its own pose loss with two distillation terms. One matches the predicted per-point features to the true DINOv2 features. The other matches the student's query embeddings to the teacher's fused query embeddings. Both terms are weighted per instance by how much the image changed the teacher's answer:

$$w_i = \min\!\big(1,\; d\big(\hat{P}^{\text{fused}}_i, \hat{P}^{\text{no-img}}_i\big)/\tau\big),$$

where $\hat{P}^{\text{fused}}_i$ and $\hat{P}^{\text{no-img}}_i$ are the teacher's pose estimates for instance $i$ with and without the image, $d$ is a symmetry-aware pose distance (rotation error plus scaled translation error), and $\tau$ is a scale hyperparameter. This adapts modality-specific distillation to structured pose. It concentrates the image-to-geometry transfer on instances where appearance actually mattered, instead of forcing the student to imitate features where they add nothing.

Two analyses accompany the system. The first is a per-slice gap study on REAL275 and HouseCat6D. Test objects are partitioned by whether the teacher's image dropout changes its answer, by category symmetry, and by depth validity in the object region (HouseCat6D's transparent and reflective items). The second is a probing test on held-out frames. It measures how well the student's predicted per-point features linearly recover DINOv2 part identity. This checks whether the student truly learned the semantics or merely learned a pose shortcut. Detection is held fixed in both: the same instance masks are given to every method, so the comparison isolates the pose network. Latency is reported both with and without the detector.

Baselines

The naive reference point is Query6DoF retrained under our exact data and schedule, with no image stream and no distillation. It reports 49.0% 5°2cm on REAL275 at 34.9 FPS on an RTX 2080Ti (Query6DoF). The student must beat it clearly at essentially the same speed, or the distillation added nothing.

The upper reference is our own DINOv2-fused teacher. The student is judged by the fraction of the teacher-minus-naive gap it closes. For context against published heavy fusion models, we also run GCE-Pose from its release with the same masks. There is no reported REAL275 number found on Lacuna for GCE-Pose, so its REAL275 numbers will be measured, not quoted.

The strongest published accuracy number in the file is CleanPose at 61.7% 5°2cm on REAL275 (CleanPose). No reported latency was found on Lacuna for CleanPose, so we time it on the same GPU and batch size. A student matching or exceeding it at ≥30 FPS would be a strong result. Being within three points of it at more than twice its measured speed would still be worth reporting.

The most dangerous baseline is a cheap-image student. It has the same query network plus a ResNet-18 crop encoder fused at projected points, trained without distillation and given the same tuning budget of an equal number of hyperparameter trials. If it matches our depth-only student at equal measured latency, the thesis that foundation semantics are what matter is weakened.

Three ablations isolate the key ingredient: output-only distillation, feature distillation with uniform $w_i$, and the full instance-weighted version. The proposed contribution is real only if the full version beats uniform weighting specifically on the slice where the teacher's image dropout changes its answer.

Why not just…?

Why not just use Query6DoF as is, since points already carry metric geometry? It is fast, but the file's evidence suggests image semantics carry real signal. DiffusionNOCS reports that DINOv2 features help resolve symmetries, and GCE-Pose's large HouseCat6D gain came from adding them. The naive baseline above measures the remaining gap directly. If distillation closes none of it, that is itself a finding that the cues are appearance-only.

Why not just add a small image CNN instead of distilling a large one? This is the real threat. A ResNet-18 on an object crop is cheap, and MultiBench's finding that simple fusion often holds its own (MultiBench) argues against assuming large backbones are needed. The cheap-image baseline at matched latency settles it. There is a further fallback: if it wins, we distill the DINOv2 teacher into that cheap-image student instead. The paper would then report the best accuracy per millisecond regardless of which stream carries it.

Why not just drop depth and distill into an RGB-only student? RGB-only category methods must recover metric scale without depth. The file reports that the unified RGB prototype method evaluates on scale-agnostic metrics and relies on iterative mesh optimization. 3D-augmented distillation was shown on orientation, not on the 2cm translation thresholds grasping needs. Depth sensors are already present on the target robots, so the proposal removes the expensive stream rather than the informative one.

Why not just distill the teacher's outputs with a standard loss? Modality-specific distillation found that plain KD trails modality-aware weighting, and ELiTe found that structured 2D-to-3D transfer needed multi-stage feature distillation. The output-only ablation tests whether this carries over to pose.

Proposed Experiments

Training uses CAMERA plus the REAL275 train split. Evaluation uses REAL275 and HouseCat6D with the standard 3D IoU at 50% and 75%, 5°2cm, 5°5cm and 10°10cm metrics. Frames per second are measured on a single RTX 2080Ti so they can be compared with Query6DoF's reported figure, and also on an embedded GPU.

Three checks decide the outcome. The first is gap closure on REAL275: the student should recover at least half of the teacher-minus-naive 5°2cm gap while staying above 30 FPS. The second is beating the cheap-image student at equal measured latency. The third is the slice analysis. Gains should concentrate on symmetric and image-dependent instances. On HouseCat6D objects with invalid depth, the student should fail where the teacher succeeds.

That last failure is a predicted result, not a nuisance. It marks the boundary where an image is worth its cost. As a follow-up, the student's own feature-prediction error could serve as a trigger for invoking the image stream only on those objects. Finally, the probing test should show that the student's per-point features recover part identity well above a points-only network's features. If they do not, the gains are shortcuts and the semantic-transfer claim fails.