Known Objects as Calibration Targets for Drifting RGB-D Sensors

Robots that estimate the 3D position and orientation of objects usually rely on two sensors: a color camera and a depth sensor. A fixed rigid transform relates the two, and it is measured once with a checkerboard. Vibration, heat and wear shift that transform over time. The best methods for estimating the pose of objects they were never trained on, called zero-shot pose estimators, quietly assume the transform is still exact: they paint image features onto depth points and then match the result against a 3D model of the object. We propose to reverse the dependency. These pose methods already receive an exact, metric CAD model for every target object. The color image alone can then fix each object's orientation and sideways position. The object's known size settles how far away it is, though only weakly. Comparing that color-only pose with the pose implied by the depth sensor gives a direct, object-level measurement of how far the camera-to-depth calibration has drifted. We pool these measurements across objects and frames into one corrected calibration, then rerun pose estimation on properly aligned data.

The idea rests on a geometric fact that is usually treated as a nuisance: monocular scale ambiguity. A single camera can tell an object's direction precisely but its distance only roughly, because a small object nearby and a large object far away look the same. This makes each object's calibration evidence lopsided. It is strong across the viewing ray and weak along it. That lopsidedness tells us how to combine evidence from objects seen along different rays, and it predicts when the calibration can be recovered and when it cannot. Both predictions can be tested.

Core Claim

We claim three things. First, current zero-shot RGB-D pose estimators lose accuracy under millimeter-to-degree-scale camera/depth misalignment, a regime nobody has measured. Second, the CAD models these methods already require can serve as free, targetless calibration rigs. Third, a calibration estimate that weights each object's evidence by its direction-dependent uncertainty recovers the drifted transform from a handful of frames. Running pose estimation on the recalibrated data should bring accuracy back near the clean-calibration level, at negligible extra cost compared with the pose estimator itself.

Where This Sits in Prior Work

The fragility of projecting points between sensors is well documented in driving and mapping. Peters and Brenner show that small calibration offsets make labels "bleed" off thin structures. SSLFusion proposes latent fusion to soften misalignment and still reports residual sensitivity. These papers motivate the proposal. The zero-shot pose methods that dominate the BOP benchmark do exactly this kind of point-level lifting and never test against it. FreeZe back-projects DINOv2 features onto depth points and registers them against CAD descriptors with RANSAC. SAM-6D matches segmented scene point clouds to CAD point clouds with a point transformer. Both treat the color and depth streams as perfectly registered. We reuse them as the pose engines whose robustness we measure and restore.

Targetless calibration methods supply the competing approach. UniCalib turns both camera and LiDAR into dense depth maps and estimates a probabilistic flow between them before solving for the extrinsic. Batch Differentiable Pose Refinement optimizes the extrinsic over learned dense features across a batch of frames. Both use the whole scene, and both need a learned depth-prediction or feature network trained on driving data. Neither uses object identity. We instead use a few strongly constrained objects whose metric shape is known exactly, and our approach requires no calibration-specific training.

On scale, DMSR shows that feeding predicted absolute depth into a pose solver corrupts rotation, and that separating geometry from scale helps. It needs category mean sizes, however. Metric3D v2 predicts metric depth from one image but needs the focal length. We borrow DMSR's lesson that scale should be separated from geometry, and we apply it in the zero-shot setting, where the CAD model supplies the exact scale that DMSR approximates with a category mean. The file's closest suggestion is its own "Gap 1", which proposes using zero-shot pose to calibrate sensors. We found no work that does it. The specific contribution here is the uncertainty-aware pooling and the observability analysis, which a naive "compare two poses" version lacks.

Research Program

The pipeline has three stages. In the first, an RGB-only pose stage estimates each detected object's pose $T_i^{c}$ in the camera frame. It uses 2D-to-3D correspondences between the image and rendered CAD templates, solved with PnP-RANSAC. The correspondences come from SAM-6D's segmentation and DINOv2 template scoring, refined with progressive pixel matching in the style of PicoPose. The stage also outputs a covariance $\Sigma_i$ from the PnP solution. This pose is metric because the CAD model is metric, but its depth along the viewing ray is uncertain. An object of physical size $L$ at distance $Z$ spans about $fL/Z$ pixels, where $f$ is the focal length. A one-pixel error in apparent size therefore shifts the estimated distance by roughly $Z^2/(fL)$, which follows from differentiating the projection. In the second stage, a depth-only registration aligns the CAD model to the depth points inside the object's mask. This gives $T_i^{d}$ in the depth-sensor frame, and it is precise along the ray but blind to the feature-lifting errors that miscalibration causes.

In the third stage, if the calibration has drifted by an unknown rigid transform $\Delta$, then $T_i^{d} \approx \Delta\, T_i^{c}$ for every object. We estimate the drift as

$$\hat\Delta = \arg\min_{\Delta} \sum_{i} \rho\!\left( r_i(\Delta)^\top \, \Sigma_i^{-1} \, r_i(\Delta) \right), \qquad r_i(\Delta) = \log\!\left( T_i^{d} \, (\Delta T_i^{c})^{-1} \right),$$

where $r_i$ is the six-dimensional rotation-and-translation residual between the two poses of object $i$, and $\rho$ is a robust loss. The robust loss handles the "hit-or-miss" failures that BOP evaluations document, where a pose estimate is either very close or completely wrong. The sum runs over objects in a sliding window of frames. Because each $\Sigma_i$ is weak along object $i$'s ray, the translation of $\Delta$ is observable only when objects are seen along several distinct rays. We will derive this condition from the summed information matrix and report it as a usable diagnostic. If the smallest eigenvalue of that matrix is too small, the system declines to update the calibration. With $\hat\Delta$ in hand, we re-register the depth to the camera and rerun FreeZe or SAM-6D unchanged.

An extension tests the scale side more directly. Commodity depth sensors also drift in depth scale, and monocular metric depth models carry scale error when the focal length is wrong. We add a global depth scale factor to $\Delta$ and substitute Metric3D v2 depth computed with a deliberately wrong focal length. This asks whether a few known objects can anchor metric scale for a monocular depth map. That is the zero-shot analogue of how LiDARTouch anchors scale with a few LiDAR beams.

Baselines

The naive reference is FreeZe and SAM-6D run unmodified on perturbed RGB-D data, with no correction. These are the strongest existing pose methods, and the file reports their clean-data numbers. FreeZe reports a mean Average Recall (AR, the BOP accuracy score averaged over error thresholds) of 69.3% on the BOP core datasets, against 63.2% for SAM-6D in the same paper's comparison. SAM-6D reports 70.4 AR on YCB-Video. No reported result on Lacuna exists for either method under calibration perturbation, which is the first gap we fill. We run both with their released settings, identical detections, and identical perturbation seeds. The clean-calibration score is the ceiling.

The strongest competing calibration approach is scene-wide targetless calibration in the spirit of UniCalib, run before FreeZe. UniCalib reports a mean error of 0.550 cm and 0.044° on KITTI LiDAR-camera calibration. No reported result on Lacuna exists for it on indoor RGB-D BOP scenes. We adapt it by using monocular depth from Metric3D v2 in place of its depth-completion network, comparing against sensor depth, and solving with the same PnP-RANSAC. It gets the same frame window as our method. The batch Gauss-Newton feature method is a second calibration baseline where its released features can be run.

The key ablation removes the uncertainty weighting: we set $\Sigma_i$ to the identity. If this does as well, the scale-aware part of the idea contributes nothing and we will report that. A second ablation estimates the drift separately for each object instead of pooling across objects, to isolate the benefit of ray diversity. We count our method as beating a baseline only if it recovers more AR under the same perturbation and frame budget, with seed variance reported.

Why not just…?

Why not just rely on the robustness the pose estimators already have? SAM-6D's background tokens and FreeZe's RANSAC already reject outlier points, so small drift may simply not matter. This is the real threat, and we test it first. If AR stays within noise of clean calibration up to realistic drift magnitudes, the proposal's motivation collapses and we will say so. Evidence from fusion detection (SSLFusion) and segmentation (Peters and Brenner) suggests that feature lifting degrades at edges first, which is where symmetry-breaking photometric cues live. That evidence comes from other tasks, though.

Why not just drop depth and use the RGB-only pose? RGB-only pose avoids miscalibration entirely, but its distance error grows as $Z^2/(fL)$, which hurts translation accuracy. FreeZe's reported +13.6% gain on TUD-L comes from a depth-based pipeline, consistent with depth adding real value. Our RGB-only stage is itself this baseline, so its AR is reported directly. If it matches recalibrated RGB-D, depth is not worth calibrating for these datasets.

Why not just run scene-wide calibration like UniCalib? It may work well. Its weaknesses are that it relies on learned depth prediction, which UniCalib identifies as its runtime bottleneck, and that dynamic or textureless regions corrupt the depth flow. Object anchors are exact metric shapes, and they are already computed for the pose task. The head-to-head comparison under an equal frame budget settles which is better, including in clutter-heavy scenes such as LM-O.

Why not just recalibrate with a checkerboard periodically? That approach is fine where downtime is acceptable. The literature documents continuous drift from vibration and wear (Liu et al.), however, and an online check that costs nothing extra also detects when recalibration is needed.

Proposed Experiments and Decisive Checks

We use the BOP datasets YCB-V, LM-O, TUD-L and T-LESS, with ground-truth poses and CAD models. We inject controlled rigid perturbations between the depth and color frames, sweeping rotation and translation magnitudes from sub-millimeter and sub-degree up to the centimeter and multi-degree scale. In the extension, we also inject depth-scale bias. We report BOP AR, calibration recovery error in millimeters and degrees, recovery error as a function of the number of frames and objects and of ray diversity, and added runtime.

Three results are decisive. The first is the sensitivity curve. Does FreeZe or SAM-6D AR fall meaningfully under realistic drift? If not, we stop. The second is observability. Does recovery error track the smallest eigenvalue of the pooled information matrix as the analysis predicts, and does the weighted estimator beat the unweighted one when objects share similar rays? The third is end-to-end recovery. Does recalibration restore AR close to the clean ceiling, and does it beat the UniCalib-style scene-wide baseline under the same frame budget? T-LESS adds a hard case, because its symmetric, textureless parts make RGB-only pose unreliable, and we expect the robust gate to reject many objects there. A small real-robot recording with a deliberately loosened depth sensor mount would test whether synthetic perturbation reflects real drift.