Defocus as a Range Sensor for the Final Inch of Wrist-Camera Grasping

A camera on a robot's wrist goes blind exactly when it matters most. In the last few centimeters before a grasp, the object comes closer than the minimum range of the depth sensor, the gripper fingers start to cover it, and the tracking software that has followed it so far loses it. Most systems respond by running the final approach without visual feedback, with the controller simply executing its last plan. When the object is moving, as on a turntable or in a human's hand, or when it is knocked during approach, the robot then grasps empty air. This proposal starts from a plain optical fact. At close range a fixed-focus lens blurs objects strongly, and the blur changes fastest precisely as the object approaches the lens. Blur is therefore not noise in the final inch. It is a range signal whose precision improves as the object gets closer, which is the opposite of how stereo and active depth sensors behave.

The idea is to treat the arm's own approach motion as a free focal sweep. Because forward kinematics, meaning the joint-angle model of where the wrist is, tells us exactly how far the camera moved between two frames, the change in blur across those frames gives metric distance without the hardware tricks usually needed to resolve defocus ambiguity. That distance then does two jobs. It keeps the grasp controller closed-loop below the depth sensor's minimum range. It also acts as a physical identity check when the target is lost and must be found again: a candidate that looks right but sits at the wrong distance, such as a duplicate part in a bin, is rejected.

Core Claim

Defocus blur of a fixed-focus wrist camera, read across frames separated by known camera displacement, yields target distance in the 2–25 cm range accurately and cheaply enough to (a) close the control loop during the final approach and (b) gate global target re-detection with a depth-consistency test. Together these raise grasp success on moving and distractor-rich targets beyond both open-loop final approach and appearance-only re-detection.

The physics that makes this work is short. From the thin-lens model in the file, an object at distance $z$ seen by a lens focused at $d_f$ with focal length $f$ and f-number $N$ produces a blur radius which, when the focus is set beyond the workspace so that $z < d_f$, can be written as

$$\sigma(z) = \kappa\left(\frac{1}{z} - \frac{1}{d_f}\right), \qquad \kappa = \frac{f^2 d_f}{N(d_f - f)}.$$

Setting the focus beyond the reach of the gripper makes every graspable object lie in front of the focal plane. That removes the front-versus-behind ambiguity that normally makes single-image defocus ill-posed, at the cost of giving up sharp imagery of the near field, which the experiments must show we can afford. Sensitivity scales as $|d\sigma/dz| = \kappa/z^2$, so a fixed blur-measurement error maps to a depth error that shrinks quadratically as the object approaches. If the camera advances by a known amount $\Delta$ between frames, the blur difference $\sigma(z-\Delta) - \sigma(z) = \kappa\left(\frac{1}{z-\Delta} - \frac{1}{z}\right)$ no longer depends on $d_f$. With a third frame, the ratio of differences also cancels $\kappa$. Metric distance then follows from blur changes and kinematics alone, which loosens the calibration rigidity that the file lists as a standing limitation of defocus depth.

Relation to Prior Work

Several papers motivate the gap. VFAS-Grasp runs a 20 Hz closed-loop grasp pipeline on a wrist RGB-D camera. It tracks with optical flow and widens its sampling radius when grasp confidence drops. Its authors report that the final approach becomes open-loop as the object leaves the view or goes under the fingers, and that proposals jump between neighboring objects because flow carries no instance identity. Active Vision Might Be All You Need restores sightlines with a third, camera-carrying arm, but the extra 7 degrees of freedom hurt simpler tasks. C3DM crops a fixation region during action denoising, and that crop lands on the occluding arm once the target is hidden. ContactGuard predicts pre-contact failures in under 20 ms but can only abort, with no way to re-acquire the target. Our system needs no extra hardware, unlike active vision. Unlike C3DM it has an explicit target-lost state. Unlike ContactGuard it supplies the recovery step after an abort, so the two are complementary rather than competing.

The re-detection literature supplies the control structure. Skimming-Perusal tracking (SPLT) splits tracking into a local verifier that declares the target absent below a confidence threshold and a fast global "skimming" classifier that proposes candidates for re-verification. Wang et al. gate model updates on correlation-peak sharpness (APCE) to avoid corrupting the appearance template. We reuse this two-state loop wholesale. What changes is the verification score: a purely 2D appearance score, which the file notes is fooled by visual twins, becomes appearance multiplied by consistency between the candidate's defocus-derived distance and the distance predicted from the target's last estimate plus camera motion. The output is a 3D target position rather than a box. RGBD1K's SPT shows that depth helps tracking. It uses stereo depth, though, which fails below roughly 20–30 cm, exactly our operating range.

The defocus literature supplies the measurement and also constrains it. Gur and Wolf learn depth by re-rendering blur through a differentiable point-spread-function layer, and Focus on Defocus shows that defocus-trained models transfer across domains because they rest on optics rather than semantics. Gur and Wolf also flag the cost of large spatially varying blur kernels, which rules out full-image re-rendering at control rates. We therefore estimate blur only on target boundaries inside the tracked or candidate region, following the boundary-blur formulation of Blurry-Edges. Focal Split resolves sign ambiguity with snapshot optical splitting. We resolve it by focus placement and get the differential measurement from arm motion instead of optics. That is cheaper, but it only works while the camera is moving, so we must test robustness when the target moves at the same time.

Research Program

We will build a Franka wrist camera with a wide-aperture, fixed-focus lens focused near 40 cm, mounted next to a conventional RGB-D sensor for comparison and ground truth at longer range. The perception module has three parts.

The first is a boundary blur estimator. It is a small network trained on rendered and real close-range sequences to output per-edge blur radius with an uncertainty. It runs only on edges of the current target region, or of each global candidate.

The second is a kinematic differential-defocus solver. It fits $z$, and when it is poorly known $\kappa$, from blur over a sliding window of three to five frames with known displacements. It also tests whether the residual is consistent with a static target. A large residual signals that the target itself moved, and that signal is passed to the controller.

The third is a two-state tracker in the SPLT style. Its local state is updated only when both appearance and depth consistency are high. Its global state skims the full frame, measures blur-derived distance for the top candidates, and accepts the one that best matches the predicted 3D position.

The controller is VFAS-Grasp's adaptive sampler, unmodified. When the RGB-D stream drops out, it is fed the defocus-derived 3D target position, so that any gain comes from the perception change alone.

We will report where this breaks. Textureless targets with weak silhouettes, full finger occlusion (defocus cannot see through an occluder, so the system can only predict and then re-detect on reappearance), and low light are the expected failure modes. Each gets its own test slice rather than being averaged away.

Baselines

The naive reference is open-loop final approach. VFAS-Grasp's pipeline runs normally until the RGB-D depth becomes invalid, then executes the last committed grasp. It shares the same robot, objects, and trials. Matching it on moving targets would mean the whole project is unnecessary.

The strongest existing system is VFAS-Grasp reimplemented with its optical-flow tracking and adaptive sampling on our hardware and objects, using the authors' settings and the same tuning budget we give our own thresholds. On its own setup it reports 80% success on a fast turntable, 81.25% on human-to-robot handover, and 84% on tabletop clearing. Those numbers come from a different setup, so the comparison that counts is our reimplementation. Beating it means a statistically significant gain, over at least 50 trials per condition, on the turntable and handover tasks and on a new twin-distractor bin task. We expect no loss on static clearing.

For re-detection, the strongest appearance-only method is SPLT, which reports an F-score of 0.616 and recall of 0.600 on VOT2018LT and a MaxGM of 0.622 on OxUvA. We will run SPLT's skimming and verification modules, fine-tuned on the same wrist-camera training sequences as ours, and plug them into the same controller.

Two ablations isolate the key ingredient. The first uses single-frame defocus with calibrated $\kappa$ and $d_f$ but no kinematic differencing. It tests whether arm motion actually buys calibration robustness, and is evaluated under deliberately perturbed focus settings. The second removes the depth-consistency term from re-detection while keeping the defocus depth for control. It tests whether depth gating, rather than depth itself, prevents locking onto the wrong instance.

Why not just…?

Why not just use a short-range depth sensor? This is the real threat. Close-range active sensors exist, and if one is clean at 2–5 cm on our objects, the defocus pipeline is mainly a cost and form-factor argument. We will include such a sensor as an upper-bound condition. The claim survives only if defocus comes within a small margin of it on success rate while degrading less on shiny objects, which the file notes break active depth for RVT-style point-cloud pipelines. If defocus does not come close, we will report that and narrow the claim to re-detection gating, which works with any depth source.

Why not just run a learned monocular depth network on the wrist image? Monocular networks infer depth from semantic layout. Focus on Defocus and Gur and Wolf give evidence that such networks transfer worse than optics-based cues. Extreme close-ups of unfamiliar objects, partly covered by fingers, are also far outside typical training data. We will run a supervised monocular depth network fine-tuned on our sequences as a drop-in replacement for the blur estimator. The decisive comparison is depth error in the 2–10 cm band, on objects held out from fine-tuning.

Why not just rely on kinematics, since the object stopped moving before it went out of range? For static targets this is probably sufficient, which is why the open-loop baseline exists. Our claim is restricted to targets that move or are disturbed during the final approach. The static-target residual of the differential solver gives a direct measurement of how often that happens in handover and turntable trials.

Why not just add a kinematic location prior to SPLT instead of depth? A 2D location prior cannot separate two identical parts lying along the same viewing ray at different depths, or a part that has slid toward or away from the camera. The no-gating ablation, run with a 2D kinematic prior added, settles this on the twin-distractor task.

Proposed Experiments and Decisive Checks

The first check is depth accuracy. On a rig where the robot touches each object with a calibrated probe to obtain ground truth, we will measure absolute distance error over 2–25 cm for textured, weakly textured, and glossy objects. We will compare defocus with kinematic differencing, single-frame defocus, monocular depth, and the short-range sensor, and run the boundary estimator alone at 20 Hz on the target crop to confirm it fits the control budget. The $1/z^2$ argument predicts that defocus error falls as distance shrinks. If it does not, the premise is wrong.

The second check is offline re-detection. We will record wrist-camera sequences of final approaches with planned disappearances (finger cover, nudges, handovers) and twin distractors. We will score them with the VOT long-term F-score and OxUvA's geometric mean of true-positive and true-negative rates, which rewards correctly saying "absent". We will compare SPLT, SPLT with a 2D prior, and our depth-gated variant. As a portability check, we will apply the same depth-consistency gate to SPT on RGBD1K, whose reported test F-score is 0.561, using its stereo depth. This tests whether gating helps beyond our own data.

The third check is closed-loop grasping. On turntable, handover, and twin-bin tasks, we will compare open-loop final approach, VFAS-Grasp, VFAS-Grasp with SPLT re-detection, the two ablations, and the short-range sensor upper bound. We will also add a recovery condition in which a ContactGuard-style abort is followed by our re-detection rather than a freeze.

The project is worth continuing if three things hold. Kinematic differencing must beat single-frame defocus under focus perturbation. Depth gating must cut wrong-instance re-acquisitions on the twin-bin task. Closed-loop success on moving targets must exceed both VFAS-Grasp and open-loop final approach. If monocular depth matches defocus in the near band, or open-loop matches us on moving targets, the central claim is false and the note should say so.