Robots that handle glassware cannot trust their depth cameras. Clear and shiny surfaces either return no depth at all or return the depth of whatever sits behind them. The most accurate transparent-object methods therefore drop depth and combine several ordinary color cameras. That move brings in a dependency nobody has measured: the cameras' relative positions must be known exactly. Wrist cameras are positioned through imprecise arm kinematics, and fixed mounts vibrate and drift. This proposal argues that the way current multi-view methods fuse their cameras makes small calibration errors damaging. It also argues that a two-stage design avoids most of the damage. In that design, each camera first finds 2D landmarks on the object, and only afterwards are the landmarks combined in 3D. The calibration can then be repaired from the transparent objects themselves, in exactly the scenes where depth-based self-calibration has nothing to work with.
In plain terms: find object keypoints separately in each image, where camera calibration plays no role. Then solve jointly for where the objects are and how far each camera's assumed position has drifted. When every camera sees several objects, the many 2D keypoint observations pin down both. We will test whether this holds pose accuracy steady under controlled calibration drift, while a method that fuses camera features early, assuming exact calibration, degrades.
Core Claim
Early-fusion multi-view methods for transparent objects warp each camera's features into a reference camera using the nominal calibration before any decision is made. MVTrans works this way: it builds a matching volume by plane-sweeping (resampling support-view features onto a stack of candidate depth planes in front of the reference camera). StereoPose does the same with epipolar losses and parallax attention, both of which match pixels along the lines that the calibration predicts. In both, a calibration error does more than shift the answer. It corrupts the matching features themselves, and the network has no way to tell that this has happened.
A two-stage design confines calibration error to the second stage, where the error is an explicit, low-dimensional quantity: six numbers per camera. The two stages are per-view 2D keypoint heatmaps followed by geometric lifting.
Our claim has three parts. First, decoupled keypoint lifting degrades more gracefully than plane-sweep fusion as extrinsic error grows. Second, the leftover reprojection error is a usable signal for estimating that calibration error. Third, the gap between the two approaches is largest for thin parts such as stems, handles, and rims. On those parts a few pixels of misprojection lands off the object entirely. This is the transparent-object counterpart of the label "bleeding" onto background that Peters and Brenner documented for poles under LiDAR-camera misalignment.
The direction this note serves is the calibration repair. It matters because existing targetless repair methods fail in this regime. UniCalib converts both sensors into dense depth maps and estimates the misalignment as flow between them, reaching 0.550 cm and 0.044° on KITTI. Its dense depth maps need valid range returns, and transparent objects supply holes. BEV-DG tolerates misalignment by pooling into coarse ground-plane columns. That kind of coarse pooling is exactly what destroys the millimeter-scale geometry a grasp needs.
Research Program
The first stage is a per-view keypoint network. It predicts category-level 2D keypoint heatmaps for each detected transparent instance: rim points, base points, handle ends, and stem junctions. Each keypoint also gets a predicted uncertainty, read from the spread of its heatmap. This stage builds on two established findings in human pose. Zhao et al. showed that features sampled around detected keypoints carry enough context for single-frame lifting. Choi et al. showed that 2D heatmaps filter out appearance variation, which here means backgrounds seen through the glass.
Transparency adds one rule to stage 1. Keypoints on the back surface are seen through the front surface, so refraction displaces their 2D positions. This is the "content aliasing" problem identified in StereoPose. Stage 1 therefore labels each keypoint as either silhouette-defined (located by the object's outline) or seen through the material. The second stage down-weights the second kind.
The second stage takes the keypoints from $V$ views and solves a robust joint problem. The unknowns are every object's pose $T_o$, every object's shape coefficients $\beta_o$ in a low-dimensional category shape model, and a per-camera correction $\Delta_v$ to the nominal extrinsics $E_v$. One reference camera is held fixed to remove the overall coordinate ambiguity. The objective is:
$$ \min_{\{T_o,\beta_o\},\{\Delta_v\}} \sum_{v,o,k} \rho\!\left( \frac{\big\| \pi_v(\Delta_v E_v T_o X_k(\beta_o)) - u_{vok} \big\|}{\sigma_{vok}} \right) + \lambda \sum_v \|\log \Delta_v\|^2 $$
The symbols are defined as follows. $X_k(\beta_o)$ is the canonical 3D position of keypoint $k$ for an object with shape $\beta_o$. $\pi_v$ is camera $v$'s projection through its intrinsics, which are assumed stable. $u_{vok}$ and $\sigma_{vok}$ are the detected 2D location and its uncertainty. $\rho$ is a robust loss that limits the influence of outliers. The weight $\lambda$ controls how strongly a prior pulls the corrections toward the nominal calibration; $\log \Delta_v$ is the size of the correction expressed as a six-number rotation and translation.
The decisive structural point is that all objects share the same $\Delta_v$. One object leaves the correction badly constrained, because moving the object and moving the camera look alike. With $O$ objects the observations number roughly $2VOK$, while the unknowns grow only as $6(V-1) + O(6 + |\beta|)$. Cluttered ClearPose scenes should therefore make the corrections identifiable. Opaque keypoints from the table plane or distractor objects can join the same objective when they are present. They are useful but not required.
A learned refinement stage, a small attention network over the lifted keypoints in the spirit of TransNet's Pointformer, then outputs size and orientation. Unlike TransNet, it takes no completed depth as input.
The analysis has three parts. We will measure how pose error and recovered-correction error grow with the magnitude of injected drift. We will characterize when the corrections become identifiable as a function of object count, view count, and view spread. We will also break errors down by keypoint thickness class, to test the thin-structure prediction.
Baselines
The naive reference is TransNet applied to each view's RGB-D image alone, with per-view estimates averaged after the nominal transforms map them into a common frame. It uses no cross-view geometry beyond that averaging, so it shows what is gained by multi-view fusion at all. TransNet reports 22.7% accuracy at 5°/5 cm on ClearPose, against 2.7% for GPV-Pose. ClearPose reports about 58% ADD-S in easy scenes and 13% under translucent covers for its baselines. Here ADD-S is the average distance from each transformed model point to the closest point of the ground-truth pose, a metric that tolerates symmetric shapes.
The strongest existing method is MVTrans, the plane-sweep multi-view network. It reports an AUC of 92.9 with five views on the real KeyPose test set, against 90.0 for stereo KeyPose and 71.9 for DenseFusion. It is the right opponent because it already beats stereo and RGB-D methods and is the clearest example of early fusion under a fixed calibration. We will retrain it on the same data, using the same view sets, view counts, and backbone family as our method and a matched hyperparameter search. Beating it means two things: matching its accuracy within noise at zero drift, and showing a significantly flatter degradation curve. The concrete target is less than half its relative AUC drop at a 1° and 1 cm perturbation.
A geometric competitor is the triangulation pipeline of Multi-View Keypoints for Reliable 6D Object Pose Estimation. It triangulates 2D keypoint heatmaps into a 3D probability map and reached about 0.5 mm and 2° on reflective metal ROBI parts. We will run it on transparent objects without its depth-based ICP refinement step, since that step needs valid depth.
The key ablation uses the identical stage 1 and stage 2 but freezes $\Delta_v$ at identity. Any gain of the full method over this ablation is attributable to the joint recalibration alone. No reported result was found on Lacuna for any transparent-object method under controlled extrinsic perturbation. That gap is the point of the study.
Why not just…?
Why not just recalibrate from the background first, then run MVTrans? This is the most serious threat. When the table and surroundings are richly textured, a standard bundle adjustment on background features may fix the extrinsics as well as we can. Two things could undercut it in glassware-dense scenes. Background features seen through glass are refracted, so their correspondences are wrong. And tabletop close-ups with a wrist camera often show little opaque texture. We will not argue this point; we will measure it. This "recalibrate then fuse" pipeline is a full baseline. We will vary the fraction of each image covered by opaque texture by cropping and masking. If background recalibration matches us everywhere, the contribution shrinks to a negative finding about early fusion, and we will report it as that.
Why not just train MVTrans with calibration-noise augmentation? Augmentation might teach the plane-sweep network to tolerate blurred matches, at some cost to precision at zero drift. We will train a noise-augmented MVTrans with the same perturbation distribution our evaluation uses, which favors that baseline. Our prediction is that augmentation trades away accuracy on thin parts. It cannot recover a drift larger than what training covered, while explicit correction can. The results broken down by thickness class and the out-of-range perturbation results will show whether that prediction holds.
Why not drop calibration entirely and learn the fusion, as CaMuViD does? CaMuViD learns projections between views and reports 95.0% detection accuracy (MODA) on Wildtrack, 9.1 points above geometric projection. But it outputs 2D pedestrian boxes, and its learned mappings are fit to a single camera layout. Grasping needs metric 6D pose, and wrist views change every frame. We will include a version adapted to output 3D boxes, as a probe of whether learned fusion transfers to camera layouts it has not seen.
Proposed Experiments
Training uses Syn-TODD multi-view renderings plus ClearPose training sequences. Evaluation uses three sources. The first is ClearPose, where the ORB-SLAM3 camera trajectories let us assemble multi-view sets of three to eight frames, including the translucent-cover and heavy-occlusion splits. The second is the KeyPose real set, whose cameras are tracked with AprilTags. The third is TOD for stereo comparison with StereoPose.
Drift is injected by perturbing the nominal extrinsics with random rotations of 0 to 3° and translations of 0 to 3 cm. The test perturbations go beyond the range the noise-augmented baseline saw in training. Metrics are AUC and ADD-S, accuracy at 5°/5 cm and 10°/5 cm, and recovered-correction error in degrees and centimeters. Pose error is also broken down by keypoint thickness class.
The decisive check is the shape of the degradation curves. The claim fails if MVTrans with augmentation or background recalibration stays within noise of our method across the whole perturbation range. A second check is whether recovered corrections track the injected ones once three or more objects are visible, and break down in single-object scenes, as the identifiability argument predicts. A third check uses ClearPose's own SLAM poses, which carry real residual error. Running the joint correction on unperturbed data should improve accuracy over nominal poses if real drift is present.
References
- MVTrans: Multi-View Perception of Transparent Objects
- StereoPose: Category-Level 6D Transparent Object Pose Estimation from Stereo Images via Back-View NOCS
- IMPROVING DEEP LEARNING BASED SEMANTIC SEGMENTATION WITH MULTI VIEW OUTLIER CORRECTION
- UniCalib: Targetless LiDAR-camera Calibration via Probabilistic Flow on Unified Depth Representations
- BEV-DG: Cross-Modal Learning under Bird's-Eye View for Domain Generalization of 3D Semantic Segmentation
- A Single 2D Pose with Context is Worth Hundreds for 3D Human Pose Estimation
- Learning to Estimate Robust 3D Human Mesh from In-the-Wild Crowded Scenes
- TransNet: Category-Level Transparent Object Pose Estimation
- ClearPose: Large-scale Transparent Object Dataset and Benchmark
- KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects
- Multi-View Keypoints for Reliable 6D Object Pose Estimation
- CaMuViD: Calibration-Free Multi-View Detection