Robot Kinematics Turn Wrist-Camera Blur into Near-Field Depth

A camera mounted on a robot's wrist is the standard fix for the arm blocking the view of the target. It has a failure of its own. In the last few centimetres before a grasp, the target fills the frame, the fingers cover part of it, depth sensors stop returning reliable values, and any fast approach smears the image. Most systems respond by slowing down, or by running the final approach open-loop. We propose to treat the smear as a measurement instead. The robot knows exactly how its camera moved during each exposure, because the joint encoders and the arm's geometric model (its "forward kinematics") give the camera's full 6-DoF pose at kilohertz rates. Removing blur normally requires guessing both the sharp image and the blur pattern, which is a famously underdetermined problem. With the camera's motion known, the only unknown left at each pixel is how far away the scene point is. Blur is strongest exactly when the target is close and the arm is fast, so it becomes a metric depth cue in the regime where every other sensor degrades.

The central claim is narrow and checkable. Known kinematics does not make blind deblurring trivial. It reduces the classical sharp-image-versus-blur-kernel ambiguity to a single scalar per pixel, inverse depth. That residual ambiguity has a predictable structure. It is worst at the point the camera is heading toward, and it disappears when consecutive frames are taken at different known velocities. A solver built around this structure should return sharp, metrically depth-annotated views of the partially occluded target at control rates. Those views should let a grasping policy keep closed-loop visual feedback while approaching fast.

Why this gap is real

Blind deconvolution recovers a sharp image $X$ and blur kernel $K$ from $Y = X * K + N$, where $N$ is noise. It is ill-posed because "sharp scene, blurry camera" and "blurry scene, still camera" ($K=\delta$, $X=Y$) both explain the data, and simple smoothness priors prefer the second answer. The literature breaks the tie in three ways. Some methods use sharp-edge priors such as the $L_0$ intensity and gradient prior of Pan et al. or the salient-edge selection of Gong et al.. Others restrict kernels to plausible families: Tran et al. learn a kernel space, and Blur2seq predicts a camera rotation sequence and refines it at test time with a re-blur loss. A third group uses auxiliary sensors: GyroDeblurNet corrects noisy smartphone gyroscopes with image features. Our setting departs from all three in the same way. Blur2seq explicitly assumes rotation only, and GyroDeblurNet's gyroscope cannot see translation. Yet a wrist camera's approach motion is dominated by translation, and translation is precisely what makes blur depend on depth.

The nearest prior work is the layered parallax-blur model of Torres and Kämäräinen. It splits the scene into depth layers, blurs each layer with a kernel derived from a known camera trajectory, and fits a coordinate-network sharp image. It takes depth as given, from an iPhone LiDAR. It does not estimate depth, and it has never been run on a robot. Our work keeps their insight that translation-induced blur is layered by depth. We invert the direction: depth becomes the quantity to be recovered from the blur, because the depth sensors are what fail at grasp range. ActiveVLA and KineDepth both document this degradation on specular, transparent and dark surfaces.

KineDepth uses kinematics differently. It tracks the robot's links to fix the scale of a monocular depth network. In our formulation scale comes for free, since the camera translation from forward kinematics is in metres. KineDepth also needs line of sight to the arm, while we need nothing beyond the joint encoders.

On the manipulation side, ContactGuard and VFAS-Grasp motivate the problem. They show that wrist cameras develop terminal blind spots where misalignment causes knocks and slips before contact can be verified. ContactGuard's answer is to predict the unseen outcome with a latent world model. That cannot correct for a disturbance that happens after vision degrades, so recovering usable observations is the complementary route. ContactGuard also constrains our design. Feeding joint states into its learned model hurt in the small-data regime because the network took proprioceptive shortcuts. We therefore use kinematics only inside an explicit image-formation model, never as an input feature. Whether this avoids the shortcut is tested directly below.

Core formulation

For a pixel $x$ outside the gripper mask $M$, the blurred observation over an exposure of length $\tau$ is

$$ Y(x) = \frac{1}{\tau}\int_0^{\tau} X\big(w_t(x;\rho(x))\big)\,dt + N(x). $$

Here $X$ is the sharp latent image at exposure start. The function $w_t$ maps $x$ to its location at time $t$ under the relative camera pose $T_0^{-1}T_t$, with each pose given by forward kinematics from the joint angles. The unknown $\rho(x)$ is metric inverse depth. The mask $M$ comes from rendering the gripper's mesh into the hand-eye-calibrated camera. Because the fingers move rigidly with the camera, they stay sharp and fixed in the image and are simply excluded from the likelihood.

During a straight approach at speed $v$, the blur length at a pixel is roughly $\rho(x)\, v\, \tau\, \lVert x - e\rVert$, where $e$ is the focus of expansion. The expression follows from first-order optical flow under forward translation. It makes the residual ambiguity explicit. A single frame cannot separate fine texture at large $\rho$ from coarser texture at smaller $\rho$, though the known blur direction does rule out the isotropic "no blur" answer. The ambiguity also gets worse toward $e$, which is usually where the target sits.

Two features of robot motion resolve it. First, the controller's own acceleration profile means consecutive frames see the same static scene points at different known velocities. Blur length then varies across frames by known factors, and only one $\rho$ explains all of them. Second, a small lateral component in the approach moves $e$ off the target and makes the centre of the target well-conditioned.

The solver has two stages. A lightweight network gives an initial estimate of $X$ and $\rho$ over a crop around the target. A short test-time refinement then minimises the multi-frame re-blur residual. It follows Blur2seq's re-blur loss, but the parameters are inverse depth over a small discrete sweep of fronto-parallel planes, in the spirit of the layered representations of Xin et al. and Torres and Kämäräinen, rather than rotation angles. A few nuisance parameters are estimated jointly, much as GyroDeblurNet refines gyro bias: a hand-eye calibration offset, an exposure timestamp offset, and rolling-shutter line delay.

Research Program

The work has three parts. The first is analysis: derive and empirically map how well inverse depth can be recovered at each pixel. The conditioning depends on texture orientation relative to the blur direction, on distance to the focus of expansion, and on how much velocity varies across frames. That map tells us when the method should work and when it cannot.

The second is the solver itself, built for a latency budget compatible with closed-loop control. GyroDeblurNet's 0.13 s per 720p image is too slow for that budget, so we work on target crops. The real-time comparison point is ContactGuard's under-20 ms monitoring loop.

The third is a small change to approach planning, inspired by visibility-aware planning such as MobileH2R. It adds a cost that keeps the focus of expansion off the predicted grasp region. This lets the robot trade a little path length for well-posed perception instead of slowing down.

Data comes from two sources. In simulation, we render physically correct blur by averaging dense sub-frames along the actual forward-kinematics trajectories of RLBench wrist cameras, at several approach speeds. On a real arm, we use the Franka Panda class used by KineDepth with a wrist RGB camera. Paired ground truth comes from robot repeatability: each trajectory is executed fast with normal exposure, then replayed slowly with short exposure to obtain sharp frames. Ground-truth depth comes from a structured-light scan taken at rest. This yields a RealBlur-style paired dataset in which the motion is exactly known, with deliberately included specular and transparent targets where depth sensors fail.

Baselines

The naive reference is the policy and perception stack fed raw blurry wrist frames at full approach speed, together with the same stack fed frames from a slow approach. These two bracket the problem. If the fast-approach success rate barely drops relative to the slow one, the motivation collapses, and we will report that.

The strongest existing deblurring competitor is Blur2seq. It reports 29.01 dB PSNR on RealBlur and 23.87 dB on the Lai nonuniform set, against a previous best of 22.11 dB. It is the strongest because it already couples restoration to physical camera motion and refines it at test time. We run it with its released training recipe and fine-tune it on our simulated blur data, with the same data and the same per-frame compute budget as our solver. A second competitor is GyroDeblurNet, which reports 27.28 dB on GyroBlur-Synth. We feed it rotations computed from forward kinematics in place of gyroscope readings. That gives it exact rotation but no translation, so it isolates whether modelling translation and depth is what matters. A third is the Torres and Kämäräinen layered model given the forward-kinematics trajectory plus depth from a commodity wrist depth sensor. This tests whether estimating depth from blur beats measuring it near the target.

The file reports no result on Lacuna for any method on eye-in-hand manipulation blur, so all of these baselines will be run fresh on our benchmark. To beat them, our solver must exceed each of them on target-crop PSNR and SSIM at matched latency. It must also beat the depth-sensor variant on inverse-depth error on specular and transparent objects. KineDepth's 0.053 m scene depth error comes from a different setup and is cited only as context, not as a target.

Two ablations isolate the key ingredient. The first is a learned deblurrer that receives forward kinematics as an input feature instead of through the forward model. This is the shortcut test motivated by ContactGuard. The second is our solver restricted to single frames, which tests whether velocity diversity across frames is what resolves the depth ambiguity.

Why not just…?

Why not just train the policy on blurry frames and let it cope? This is the most serious threat. Modern policies can absorb a lot of nuisance variation. On the other side, THE COLOSSEUM reports 30–50% success drops under visual perturbations, and Parés-Morlans et al. find that VLAs fail at fine contact alignment. Neither tested motion blur specifically. We will train the same policy architecture on the same demonstrations with physically rendered blur augmentation and compare success across approach speeds. If the augmented policy matches ours, the honest conclusion is that explicit inversion is unnecessary for control. Its remaining value would then be limited to metric depth for monitoring.

Why not just slow down before contact? It works, and it is the default. Its cost is cycle time, and slowing does not remove the finger occlusion itself. We will plot grasp success against approach speed for slowing and for our method. If success stays flat down to modest slowdowns, the method buys little. If our method holds success at speeds where raw frames fail, that measures what it buys.

Why not just use a shorter exposure or a high-speed camera? At a fixed light level, shorter exposure trades blur for noise. Wider apertures shrink depth of field at grasp range, as the dual-pixel work of Xin et al. illustrates, and high-speed systems are bulky, per Zhang et al.. We will include a short-exposure arm at matched photon budget, so the comparison is noise against blur rather than a free lunch.

Why not just use the wrist depth sensor? Near-field, specular and transparent targets are exactly where ActiveVLA and KineDepth report degradation. The depth-sensor variant of the layered model is our direct test of this alternative.

Decisive Checks

The method succeeds if four things hold. Inverse-depth error should follow the predicted conditioning map: high near the focus of expansion on single frames, and falling when velocity diversity or a lateral approach component is added. The solver should beat Blur2seq and FK-fed GyroDeblurNet on target-crop restoration at control-compatible latency. A ContactGuard-style pre-contact failure probe on restored frames should not lose ROC AUC as approach speed rises. ContactGuard reports 0.992 for Cup and 0.946 for Box on slower real executions. Finally, grasp success on the real arm and on blurred RLBench should stay higher than both the slowed and the blur-augmented baselines at high speed.

If the kinematics-as-feature ablation matches the forward-model version, our explanation for ContactGuard's proprioception failure is wrong, and we will report that result.