The Robot Is the Checkerboard: Self-Calibrating Intrinsics During Sim-to-Real Adaptation

Robots already see their own arm in almost every camera frame. The arm is a known 3D shape whose joint angles are reported exactly at every instant. That makes it a moving calibration target, much like a checkerboard waved in front of the lens, except that nobody has to wave it. This proposal builds a camera-to-robot calibration system that is trained purely in simulation and adapted to real video without labels. It recovers the camera's internal optics (focal length, image center, lens distortion) together with the camera's position and orientation relative to the robot. Existing markerless methods recover only position and orientation, and they assume the internal optics were measured beforehand by hand.

This matters because the best current sim-to-real geometry methods depend on a calibration they do not produce. CtRNet adapts a simulation-trained keypoint and silhouette network to real images by checking the two against each other through a differentiable renderer, but it takes the camera matrix as given. AnyCamVLA re-renders arbitrary test cameras into the viewpoint a frozen robot policy was trained on, but only when the test camera's parameters are known exactly. A bumped camera, a changed zoom, or a swapped lens breaks both. We ask whether the same self-supervised consistency signal that closes the appearance gap can also estimate the optics, and whether estimating the two jointly beats doing them in sequence.

Core Claim

In CtRNet-style self-supervised adaptation, an error in the assumed optics is not harmless. The silhouette rendered through a wrong camera model disagrees with the true silhouette in a systematic way, and the adaptation loop "fixes" this by teaching the keypoint and segmentation networks to be wrong. Our claim is that the fix is to treat the optics as shared unknowns during adaptation. Concretely, one set of intrinsics $\theta = (f, c_x, c_y, k_1, k_2)$ is shared by all frames of a static camera, and one camera-to-robot pose is estimated per frame or per clip. Both are optimized alongside the networks, and the robot's joint telemetry supplies many distinct known 3D configurations. Each configuration contributes 2D–3D constraints on $\theta$. This is the classical reason multi-pose targets make intrinsics observable when a single view cannot, a single-frame ill-posedness the file documents for scene-based methods.

The objective sums, over real frames $t$, a silhouette consistency term and a keypoint reprojection term:

$$\min_{\phi,\ \theta,\ \{T_t\}} \sum_t \Big[ \mathcal{L}_{\text{sil}}\big(R(q_t, T_t, \theta),\ S_\phi(I_t)\big) + \lambda\, \big\| \pi_\theta(T_t\, X(q_t)) - K_\phi(I_t) \big\|^2 \Big]$$

Here $I_t$ is the image, $q_t$ the measured joint angles, and $X(q_t)$ the 3D keypoints from forward kinematics. $T_t$ is the camera-to-robot pose, and $\pi_\theta$ is projection with distortion. $R$ is a differentiable silhouette renderer that uses the same distorted projection. $S_\phi$ and $K_\phi$ are the network's segmentation and keypoint heads, and $\lambda$ weights the two terms. $T_t$ is produced by a differentiable pose solver given the current $\theta$, with gradients to $\theta$ taken through the implicit function theorem as CtRNet does for pose. The new element is that $\theta$ receives gradient and is shared across time. The network cannot absorb a per-frame optics error, because $\theta$ must explain every arm configuration at once.

Research Program

We reuse CtRNet's pipeline, a simulation-pretrained backbone, a differentiable pose solver, and a silhouette renderer. We change three things. First, the projection and renderer carry a radial distortion model, starting with a two-coefficient polynomial and optionally the generic third-order form of Wakai et al. for wide lenses. Second, simulation pretraining randomizes intrinsics and distortion, so the network does not learn a shortcut tied to one fixed virtual lens. Third, $\theta$ is initialized from a crude guess, such as the manufacturer's nominal focal length with zero distortion, or from a closed-form multi-frame solve. It is then refined jointly.

We then analyze observability. Focal length trades off against depth along the optical axis, which is the same entanglement the file flags for single-view regressors and for CalibNet's weak Z translation. We compute the rank and conditioning of the Jacobian of reprojected keypoints with respect to $(\theta, T)$ as a function of how much the arm moves in depth and across the image. This yields a practical rule for how many and which joint configurations are needed. It also supports a small active variant: the robot executes a short motion script chosen greedily to maximize the smallest eigenvalue of that information matrix. That turns recalibration into a thirty-second self-check rather than a technician visit.

Finally, we connect the result to downstream use by feeding the recovered parameters into AnyCamVLA's view-synthesis front end. This tests whether blind recalibration is good enough for a policy that was never trained on the new camera.

Baselines

The naive reference is CtRNet run exactly as published but given the wrong intrinsics a real deployment would have: nominal focal length and zero distortion after a simulated bump, zoom change, or lens swap. The upper reference is CtRNet with the true, checkerboard-measured intrinsics. On DREAM-real, CtRNet reports 85.9% AUC on the average-distance (ADD) pose metric in this setting, against roughly 80% for RoboPose and roughly 68% for DREAM. That 85.9% is the number to approach, since our method does not see the true intrinsics. We found no reported result on Lacuna for CtRNet under perturbed intrinsics, so measuring that degradation curve is itself our first experiment. A method "wins" if it recovers most of the gap between the wrong-intrinsics run and the 85.9% oracle.

The strongest competitor is a two-stage pipeline built from the same parts. Stage one is CtRNet adaptation with nominal intrinsics. Stage two takes its keypoints across all frames, pairs them with forward-kinematics 3D points, and fits intrinsics, distortion, and poses by robust bundle adjustment. RANSAC-initialized minimal solves with the P5Pfr solver supply the starting point, since it recovers pose, focal length, and one radial term from five correspondences in about 2 microseconds. This is essentially classical calibration with the robot as the target. It is the most likely way our idea could turn out unnecessary. Both pipelines get the same real frames, the same pretrained network, the same number of adaptation steps, and the same distortion model. Beating it means lower focal and distortion error and higher ADD AUC on held-out frames, with the margin largest when the initial intrinsics are far off.

Our ablations each remove one element. Joint optimization without intrinsics randomization in simulation isolates the pretraining change. Optimizing $\theta$ per frame instead of sharing it tests whether sharing is what prevents absorption. Keypoint-only and silhouette-only variants show which signal carries the optics information. A third baseline, for scene-based single-image calibration, runs the Manhattan-world heatmap estimator of Wakai et al. on the workspace image and then CtRNet with the resulting intrinsics. It gets the same frames and no extra tuning.

Why not just…?

Why not just calibrate with a checkerboard once? In a single lab this is cheap, and we do not pretend otherwise. The case for our method is recalibration after the fact. Bumped cameras, variable-zoom cameras, and fleets of robots with cheap lenses all need it, and the file documents drift and marker maintenance as recurring costs for markers on robots (CtRNet) and for camera domes (Multi-Cali Anything). We therefore evaluate against checkerboard intrinsics as ground truth, not as a competitor. The honest success criterion is matching a checkerboard to within a margin that does not hurt downstream pose or policy success.

Why not just adapt with nominal intrinsics, then run bundle adjustment on the keypoints? This is the real threat. If the network's keypoints stay unbiased under wrong intrinsics, the two-stage pipeline should do about as well as ours. Our hypothesis is that silhouette self-supervision under a wrong camera model biases the keypoints toward the wrong projection, and that a second stage cannot undo that bias. The decisive experiment sweeps the initial focal error and distortion strength and compares the two pipelines at each level. If they tie across the sweep, the contribution shrinks to "use the robot as a target with a sim-trained keypoint detector," and we will report it as such.

Why not just use a feed-forward video calibrator such as AnyCam? AnyCam recovers intrinsics and poses from casual video in about 5 seconds. Its signal, though, comes from camera motion relative to a scene, and the workspace camera we target is static, so there is little motion parallax to exploit. Scene-based single-image calibrators (Wakai et al.) lean on Manhattan-world structure that tabletops only partly provide. We include the latter as a baseline and report AnyCam on the static clips so the comparison is shown rather than assumed.

Why not just randomize intrinsics in simulation and regress them directly? Direct regression discards geometric consistency. The file notes that black-box regression gives reprojection errors above 100 pixels where curvature-constrained estimation gives 0.47 (Xue et al.). We run a regression head trained with the same randomization as a baseline.

Proposed Experiments

We start from DREAM-real Panda sequences, whose true intrinsics are known, and the Baxter set used by CtRNet. We hide the intrinsics and initialize from the nominal value perturbed by ±10–30% focal error and a shifted image center. To test distortion with ground truth, we apply known radial warps to the real images. This is a partial measure, since the warp is synthetic even though the image content is real, and we say so. We therefore also record new sequences of a Panda through two physical lenses, one near-pinhole and one wide-angle, with checkerboard intrinsics as ground truth. Metrics are ADD AUC for pose, relative focal error, image-center error in pixels, distortion-profile error, and held-out reprojection error.

The first decisive check is the degradation curve of published CtRNet under wrong intrinsics. If the ADD AUC barely drops, the motivating problem is small. The second check is joint versus two-stage across the error sweep, as described above. The third is observability: accuracy as a function of arm motion range, with and without the active motion script. We expect focal error to rise sharply when the arm stays at nearly constant depth, which would confirm the predicted focal-depth ambiguity. The fourth is downstream. On LIBERO, where AnyCamVLA reports 94.5% success on unseen viewpoints with known parameters against 67.9% for the base policy, we perturb the camera without telling the system. We then compare success with our recovered parameters, with the stale original parameters, and with the true ones. Recovering most of the distance from stale to true would show that blind self-calibration closes the loop that current view-adaptation methods leave open.