Silent Keypoint Errors: How Receptive Field Shapes Pose Error Propagation

Many robot pipelines estimate an object's 3D position and orientation (its "6D pose") in two steps. A neural network first finds a set of 2D keypoints, meaning named points such as corners or holes, in the image. A geometric solver then fits the rigid pose that best explains those keypoints. The receptive field of the keypoint network, meaning how much of the image each output pixel can see, is usually discussed only as an accuracy knob, with the assumption that more global context is better. We argue it matters for a second reason. It decides the shape of the errors, not just their size. A detector that looks only locally makes errors that are roughly independent across keypoints. The geometric solver can see such errors, because they break the object's rigid shape, and it partly averages them away. A detector with global context makes coordinated errors: the whole constellation of keypoints shifts together as if the object had moved. The solver cannot tell these errors apart from a real motion, so they pass straight into the pose and then into the robot's grasp or insertion. They also leave no reprojection residual that would warn anyone.

This matters because downstream manipulation is limited mainly by pose accuracy, especially translation. In Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success, 18.8 million simulated grasps show that 6D pose error, dominated by translation, bounds grasp success more than mesh fidelity does. It also matters because the best distribution-free uncertainty method for keypoint pipelines treats keypoint errors as separate per-keypoint boxes. That method, SLUE in Uncertainty Quantification for Visual Object Pose Estimation, uses conformal prediction, a calibration procedure that gives coverage guarantees without distributional assumptions. We propose to measure how receptive field changes the geometry of keypoint error. We then build a detector loss and a joint calibration procedure that target the error component the solver cannot see.

Core Claim

Stack the $K$ keypoint errors of one image into a vector $e \in \mathbb{R}^{2K}$. Let $J \in \mathbb{R}^{2K \times 6}$ be the Jacobian of the keypoint projections with respect to the six pose parameters, evaluated at the true pose. To first order, a least-squares pose solver returns the pose error $\delta = (J^\top J)^{-1} J^\top e$. It therefore responds only to the part of $e$ that lies in the column space of $J$:

$$ e = P e + (I - P) e, \qquad P = J (J^\top J)^{-1} J^\top . $$

We call $Pe$ the pose-equivalent error. It is invisible to the solver and becomes pose error. We call $(I-P)e$ the shape-inconsistent error. It is visible as reprojection residual and has no first-order effect on the pose. The quantity we track is the silent fraction $\rho = \|Pe\|^2 / \|e\|^2$. If errors were independent and isotropic, the expected $\rho$ would be $6/(2K)$, because a random vector puts that share of its energy into a fixed 6-dimensional subspace of a $2K$-dimensional space. For 8 keypoints this is 0.375.

Our hypothesis has two parts. First, local convolutional detectors sit near this independent baseline but have heavy-tailed outliers. Second, detectors with global routing reduce outliers but push $\rho$ well above $6/(2K)$. If so, a larger receptive field can lower keypoint error while improving pose error less than expected. It can also make calibration that treats keypoints independently either loose, via union bounds, or quietly miscalibrated when conditioned on residuals. The new contribution is to treat the receptive field as a knob on error geometry, and to design training and calibration around the silent component.

Relation to Prior Work

SLUE (Uncertainty Quantification for Visual Object Pose Estimation) supplies our method. It builds conformal 2D boxes per keypoint and turns them, through an S-lemma convex relaxation, into a minimum-volume 3D ellipsoid that contains the true pose with high probability. Its reported weaknesses are hallucinated outlier keypoints and symmetric objects. We keep its ellipsoid machinery and replace its per-keypoint calibration with a joint score split along $P$. SLUE never asks how the detector architecture shapes cross-keypoint correlation, and that question is our main one.

The global-routing literature supplies the architectures we vary. CCNet spreads context along rows and columns at $O(N\sqrt{N})$ cost, with 11x less memory than dense non-local blocks. Broadcasting Convolutional Network appends explicit coordinate channels and broadcasts a pooled global vector, and it reports a sixfold reduction in localization error on Scaled-MNIST. Compact Generalized Non-local Network couples positions and channels in linear time. All three are evaluated only on segmentation, classification, or relational question answering, where how errors are correlated does not matter. We use them as controlled points along a receptive-field axis.

Among manipulation systems, Fast robust peg-in-hole insertion with continuous visual servoing is our local reference. It uses a ResNet-18 U-Net heatmap detector and treats heatmap peaks deterministically. TAX-Pose uses cross-attention and then a weighted SVD solve. That is a global-context correspondence model feeding a solver, the regime where we expect silent errors, and it outputs no uncertainty. FoundationPose avoids keypoints through render-and-compare refinement and ranking transformers, and it produces no certifiable bounds. Deep learning a grasp function for grasping under gripper pose uncertainty motivates the downstream use. Blurring a grasp-quality map with a pose-noise kernel improves robustness, but only if the kernel reflects the real error shape, which our analysis provides.

Research Program

The work has three stages.

The first stage is measurement. We train keypoint detectors that share one decoder, one keypoint set, and one training data regime on LineMOD-Occlusion and YCB-Video. They differ only in the context mechanism: a plain ResNet-18 U-Net, the same network with CCNet blocks, with coordinate channels and broadcasting, with CGNL blocks, and a ViT encoder. We match compute across them roughly. For each model we compute $\rho$ per image from ground-truth poses. We also compute the covariance of $e$ in the object frame, how often outliers occur, and how $\|Pe\|$ relates to the observable residual $\|(I-P)e\|$. We then vary occlusion level, keypoint count, and object symmetry to see where coordinated errors appear.

The second stage is a training intervention. We add a loss term that weights the pose-equivalent component more heavily, $\mathcal{L} = \|e\|^2 + \lambda \|Pe\|^2$, computed with the ground-truth Jacobian during training. This asks a global detector to keep its outlier robustness while spending its capacity on the component that reaches the pose.

The third stage is joint calibration. We replace per-keypoint conformal boxes with a two-part score. The observable residual part stays as a per-image covariate. The silent part is calibrated directly, as a Mahalanobis-type score on $Pe$ fitted on held-out data and grouped by residual magnitude. The result feeds a pose ellipsoid that is either SLUE-style or computed directly. Finally, we propagate the resulting pose errors and ellipsoids into grasp execution and blurred grasp-quality planning.

Baselines

The naive reference is the local detector used in continuous servoing (Fast robust peg-in-hole insertion with continuous visual servoing): a ResNet-18 U-Net with a least-squares pose solver and SLUE per-keypoint calibration. Every other variant must beat it on pose error and on ellipsoid volume at matched coverage. For SLUE the file reports "90% coverage for a 90% confidence interval" and about 39 ms on CPU. Any calibration variant must reach 90% empirical coverage at the 90% target and add comparable or smaller overhead. No reported result was found on Lacuna for SLUE's absolute ellipsoid volumes on LM-O or YCB-V, so we rerun SLUE ourselves on identical detector outputs.

The same local detector with robust, RANSAC-style pose fitting is the cheapest fix for hallucinated keypoints. It is the most direct competitor on the outlier failure mode. It gets the same detector weights and a tuned inlier threshold.

The global backbones (CCNet, coordinate broadcasting, CGNL, ViT) trained with the plain keypoint loss form the ablation for the backbone swap without our loss. The same local backbone trained with the silent-component loss forms the ablation for the loss without the swap. All models share training images, augmentation, number of epochs, and a fixed hyperparameter search of equal size. Their inference FLOPs are reported so that gains are not bought with compute.

The strongest existing method for downstream success is FoundationPose. In Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success it reaches 89.9% average grasp success, against 59.4% for MegaPose. We run it on the same YCB objects and grippers in that PyBullet protocol. A certified keypoint pipeline counts as competitive if it comes within a few points of 89.9% while also giving calibrated bounds and lower latency than FoundationPose's reported 1.3 s initialization. It counts as beating FoundationPose if uncertainty-aware grasp selection, using the planner blurred by our ellipsoids, exceeds 89.9%.

Why not just…?

Why not just reject outliers with a robust solver? Robust fitting deals with the hallucinated keypoints that SLUE names as its main weakness, and it may close most of the gap for local detectors. It is a real threat to the "use global context" half of any proposal. By construction, though, it cannot remove pose-equivalent error: a keypoint set that shifts coherently is a perfect inlier set. The measurement stage settles the question. If robust fitting on the local detector matches the global detectors on translation error and ellipsoid volume, the architectural half of the proposal is unnecessary and the contribution reduces to the calibration analysis. We will report that outcome if it occurs.

Why not just use FoundationPose? It already posts 89.9% grasp success in the benchmark paper, so for raw success it may simply win. It gives only ranked hypotheses with no coverage guarantee, and its 1.3 s initialization is a limit in closed-loop use. Our claim is not that keypoint pipelines beat it on accuracy. It is that a keypoint pipeline with silent-error-aware calibration offers a speed and certification tradeoff that FoundationPose does not. The grasp benchmark comparison measures how much accuracy that tradeoff costs.

Why not just swap in a bigger or global backbone? If global context lowered keypoint error without raising $\rho$, the backbone swap alone would be the answer and our loss and calibration would be redundant. The backbone-only ablation tests this directly. The proposal stands only if global backbones show $\rho$ clearly above $6/(2K)$ and if the silent-component loss or joint calibration improves pose error or bound tightness beyond the backbone swap alone.

Why not keep SLUE's per-keypoint boxes? When errors are independent, per-keypoint boxes are nearly right. When errors are coordinated, a union of per-keypoint boxes wastes volume on shape-inconsistent directions the solver already filters out. It also gives no sign that small residuals can hide large pose errors. Comparing ellipsoid volume at matched coverage, and coverage conditioned on low residual, will show whether this matters in practice.

Proposed Experiments

The first decisive check tests the hypothesis itself. On LM-O and YCB-V, does the silent fraction $\rho$ rise with receptive field, from near $6/(2K)$ for the U-Net to substantially above it for CCNet, CGNL, and ViT? And does it rise faster under occlusion? If $\rho$ stays flat across architectures, the central claim is false.

The second check asks whether that rise matters. We compare the ratio of pose translation error to mean keypoint error across detectors. We also test whether coverage conditioned on small observed residual falls below the nominal 90% for global detectors under per-keypoint SLUE calibration.

The third check measures the interventions. We report ellipsoid volume at 90% coverage and ADD-0.1d with and without the silent-component loss and the joint calibration.

The fourth check covers downstream effects. We run the PyBullet grasping protocol of Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success on its 21 YCB objects and 9 grippers. We compare grasp selection on point estimates with grasp selection over our calibrated ellipsoids, against FoundationPose's 89.9%. As a secondary, precision-driven check, we inject each detector's error distribution into a Nut Assembly insertion in robosuite and measure success as a function of clearance.

Terminal self-occlusion during insertion is out of scope. There, vision must hand off to touch, as in Safely Learning Visuo-Tactile Feedback Policies in Real For Industrial Insertion. Our results apply to the approach phase that sets the error budget touch must then absorb.