A mobile robot reaching for a drawer handle has no lab motion-capture system telling it where it is. It knows its position only by adding up small wheel and camera motion estimates, a process called dead reckoning, and those estimates drift. In the last few centimeters before contact, its own gripper also blocks the wrist camera's view of the handle. These two failures are usually studied separately, but they are the same problem. In both cases the robot moves for some stretch of time with no fresh visual fix on the thing it cares about, and the error it accumulates over that blind stretch decides whether the grasp succeeds. This proposal treats the blind stretch as a quantity that the controller can measure and budget. The slow, high-level planner states goals relative to the target object rather than in a global map. The fast, low-level controller keeps an uncertainty estimate of where the object is relative to the gripper. The system follows one rule: stop moving the base before the last visual fix, so that every motion during the blind period comes from the arm, whose joint encoders drift far less than wheels do.
The opportunity is practical. The strongest uninstrumented precision system in the file, A Training-Free Framework for Precise Mobile Manipulation of Small Everyday Objects (SVM), servos the base and arm directly on the target with a wrist camera. It removes the gripper from the image with a video inpainting model and reaches 71% success on 72 unseen everyday objects, against 29% for open-loop execution. The inpainting and foundation-model stack, however, runs at about 0.1 Hz. Our claim is that most of what inpainting buys can be obtained cheaply. The robot needs to control when it goes blind and how much it moves while blind, not to see through its own fingers. Done that way, the controller should match SVM's precision at above 10 Hz.
Core Claim
The terminal positioning error of an uninstrumented mobile manipulator is governed by the uncertainty it accumulates between its last reliable observation of the target and the moment of contact. Let $\Sigma_{\text{fix}}$ be the covariance of the gripper-to-object pose estimate at the last good visual fix, $t_b$ the length of the blind interval, and $Q_{\text{arm}}$ and $Q_{\text{base}}$ the per-second noise added by arm kinematics and base odometry. The predicted covariance at contact is then approximately
$$\Sigma_{c} \approx \Sigma_{\text{fix}} + t_b\,(Q_{\text{arm}} + \alpha\,Q_{\text{base}}),$$
where $\alpha \in [0,1]$ is the fraction of blind-interval motion carried by the base. This is the prediction step of a Kalman filter with constant process noise. Wheel slip and heading error make $Q_{\text{base}}$ much larger than $Q_{\text{arm}}$, so the controller can shrink $\Sigma_c$ in three ways. It can drive $\alpha$ to zero by parking the base before the view is lost. It can place the last fix as close to contact as the geometry allows, which shortens $t_b$. It can refuse to commit when the standard deviation of $\Sigma_c$ along the task's critical axis exceeds the tolerance $\tau$ of the subgoal, for example half the clearance of a handle. In that case it backs off and re-observes instead of pushing forward. The new element is a two-way interface between the planner and the controller. The planner sends down an object-anchored subgoal with a tolerance. The controller sends up a predicted contact uncertainty. The commit decision depends on both.
Relation to Prior Work
Several systems motivate the gap. Global-frame pipelines degrade in exactly the way the claim predicts. The HomeRobot open-vocabulary mobile manipulation benchmark runs a modular skill chain on a Stretch robot and reports 20% real-world success; its authors attribute part of the downstream grasp failure to base dead-reckoning error. SVM showed that servoing on the target, rather than navigating to a map coordinate, recovers most of the lost precision. It never models the blind interval, though, and pays for occlusion handling with latency. GHIL-Glue and SuSIE filter or generate visual subgoals by checking whether they look plausible. They never ask whether the subgoal can be reached given the robot's current uncertainty, and our tolerance-versus-covariance gate supplies that missing check.
Other systems supply the method. AdaptManip is the closest prior system. It fuses intermittent vision with proprioception in a recurrent object-state estimator on a humanoid and uses LiDAR for global position. It depends on an AprilTag on the box and on a fixed navigate–lift–deliver script. It also never reasons about how base motion during occlusion inflates object uncertainty, or uses that estimate to decide when to commit. Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions shows on a fixed arm that passing explicit uncertainty to the controller pays off. It reports 94% success under frequent occlusion against 12% for end-to-end reinforcement learning, a fixed-arm result we extend to a moving base. Aerial Manipulation with Contact-Aware Onboard Perception and Hybrid Control shows that contact itself is a measurement: adding a zero-normal-velocity factor cut contact velocity error to 0.0121 m/s, against 0.0619 m/s for OpenVINS. We borrow this idea to reset object-relative uncertainty at first touch.
Two papers constrain the design. ContactGuard found that feeding joint states into a learned world model hurt failure prediction, because the model used them as a shortcut. We therefore use proprioception only in an analytic filter prediction step and never as a learned input, which sidesteps that failure mode. HiRT shows why rate matters: its decoupled 10 Hz executor raised dynamic-task success from 48% to 75% over roughly 4 Hz monolithic models. That result is why we treat SVM's 0.1 Hz as a real weakness rather than a cosmetic one.
Research Program
We would build the system on a Hello Robot Stretch with a wrist RGB-D camera, the platform used by HomeRobot, SVM, and Harmonic Mobile Manipulation.
The slow layer runs at about 1 Hz. It uses an open-vocabulary detector (Detic, as in SVM) to pick the target. It emits a subgoal expressed as an image keypoint on the object, a contact offset, and a task tolerance $\tau$ taken from the detected part's extent. It contains no global coordinates.
The fast layer runs at 15 Hz or more. It keeps a filter over the gripper-to-object pose. Its prediction step uses arm forward kinematics and wheel odometry, with $Q_{\text{base}}$ and $Q_{\text{arm}}$ calibrated once per robot. Its update step uses a lightweight point tracker on the wrist image. Rather than inpainting the gripper, we mask its pixels using the known kinematic projection, which costs almost nothing.
A view-planning step chooses a pre-contact pose with two properties: the base can be parked there, and the arm alone can finish the approach. The step prefers poses whose final fix comes latest along the approach. At first contact, a contact factor resets the filter. When the predicted contact uncertainty exceeds $\tau$, the robot retreats to a viewing pose and the planner is asked for a fresh subgoal.
In parallel, we run a controlled study in simulation. We inject known base drift rates and occlusion onsets into ProcTHOR scenes from the Harmonic Mobile Manipulation task suite, using doors and fridges with articulated handles. This tests whether the measured terminal error follows the linear-in-$t_b$ form above.
Baselines
The naive reference point is open-loop execution: localize the target once, navigate, then execute without visual feedback. SVM reports 29% success for this on its 72-object real-world protocol, and we will rerun it on our object set.
The strongest existing method is SVM itself. Its whole-body servoing with ProPainter inpainting, Detic, and CoTracker reports 71% overall zero-shot success and 62% with inpainting removed. We will run the released pipeline on the same robot, the same object instances and environments, and the same GPU. We will match the number of attempts per object and use identical starting-distance distributions. A win means matching SVM's success within the confidence interval at ten times or more its control rate, or exceeding it on targets smaller than the gripper fingers, where occlusion is worst.
A third baseline is a global-frame hierarchy in the HomeRobot style. It maps the scene, places the subgoal in map coordinates, and grasps on arrival. Its reported real-world success is 20% on full open-vocabulary mobile manipulation tasks, but that number covers a different task. We will rerun it on our targets rather than compare across tasks.
For an end-to-end reference, we will train a Harmonic-style policy on our door and handle tasks with the same simulation budget. Harmonic reports 70% on real door pushing. There is no reported result found on Lacuna for it on small-target precision tasks.
Three ablations isolate the key ingredient. The first lets the base move during the blind interval, setting $\alpha$ to its natural value. The second removes the uncertainty gate. The third replaces masking with SVM's inpainting while keeping everything else fixed.
Why not just…?
Why not just keep SVM's inpainting and run it on a faster GPU? Inpainting answers the wrong question. It recovers the image around the fingers, but once the target itself is under the fingers there is nothing left to recover, and the robot is blind anyway. The masking-versus-inpainting ablation measures how much of SVM's 9-point inpainting gain survives when blind-interval motion is controlled. If inpainting still adds a lot at a matched rate, our cheap masking is insufficient, and we would report that.
Why not just mask the gripper and servo continuously, without any covariance machinery? This is the real threat. It may capture most of the benefit by itself. The baseline that combines masking with continuous servoing, the no-gate ablation, and the moving-base ablation are designed to settle the question. Our claim survives only if success falls measurably as injected base drift grows for continuous servoing but not for the base-freeze-plus-gate system.
Why not just fix drift with good SLAM and loop closure? Fetch-style tasks rarely revisit places, so loop closure seldom fires. HomeRobot's global-frame pipeline reaches only 20% in a real home. Millimeter-scale targets also need precision that a room-scale map does not deliver. The drift-injection sweep tests this directly by measuring the global-frame baseline's success against drift rate.
Why not just train an end-to-end policy that learns drift and occlusion implicitly? Harmonic Mobile Manipulation shows this works for large articulated targets. The visuotactile occlusion study, however, found end-to-end reinforcement learning at 12% under frequent occlusion, against 94% with explicit uncertainty. Our end-to-end baseline checks whether that gap carries over to mobile small-target grasping.
Decisive Experiments
The central test is the drift-injection sweep in simulation. We vary $Q_{\text{base}}$ over an order of magnitude and the occlusion onset distance over 2 to 15 cm, then plot terminal error and success. The claim predicts three things. First, terminal error grows linearly in $t_b$, with slope set by $\alpha$. Second, the base-freeze system's success curve stays flat as drift grows while every baseline that moves the base during blindness declines. Third, the uncertainty gate's retreat decisions catch most would-be failures. We score the third prediction as area under the ROC curve for "predicted contact uncertainty above $\tau$" against actual failure. ContactGuard's 0.946 to 0.992 on fixed-arm tasks is a reference for that metric, though its tasks differ from ours.
On hardware, we repeat SVM's protocol on knobs, switches, and drawer handles in unseen buildings. We report success, control rate, wall-clock time, and the number of retreats.
The idea is falsified if two things hold together: base-freeze gives no gain over continuous whole-body servoing at matched drift, and masking-only servoing already matches SVM. In that case the blind interval does not matter, and the simpler system wins.
References
- A Training-Free Framework for Precise Mobile Manipulation of Small Everyday Objects
- HomeRobot: Open-Vocabulary Mobile Manipulation
- GHIL-Glue: Hierarchical Control with Filtered Subgoal Images
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models
- AdaptManip: Learning Adaptive Whole-Body Object Lifting and Delivery with Online Recurrent State Estimation
- Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions
- Aerial Manipulation with Contact-Aware Onboard Perception and Hybrid Control
- ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Models
- HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers
- Harmonic Mobile Manipulation