A Training-Free Framework for Precise Mobile Manipulation of Small Everyday Objects

Arjun Gupta, Rishik Sathua, Saurabh Gupta

Abstract

Many everyday mobile manipulation tasks require precise interaction with small objects, such as grasping a knob to open a cabinet or pressing a light switch. In this paper, we develop Servoing with Vision Models (SVM), a closed-loop training-free framework that enables a mobile manipulator to tackle such precise tasks involving the manipulation of small objects. SVM employs an RGB-D wrist camera and uses visual servoing for control. Our novelty lies in the use of state-of-the-art vision models to reliably compute 3D targets from the wrist image for diverse tasks and under occlusion due to the end-effector. To mitigate occlusion artifacts, we employ vision models to out-paint the end-effector thereby significantly enhancing target localization. We demonstrate that aided by out-painting methods, open-vocabulary object detectors can serve as a drop-in module to identify semantic targets (e.g. knobs) and point tracking methods can reliably track interaction sites indicated by user clicks. This training-free method obtains an 85% zero-shot success rate on manipulating unseen objects in novel environments in the real world, outperforming an open-loop control method and an imitation learning baseline trained on 1000+ demonstrations by an absolute success rate of 50%.

Introduction

Mobile manipulation in unstructured human environments presents a significant challenge for robotics. While navigation to a general area like a kitchen is increasingly robust, the final stage of interaction—the "last few centimeters"—requires extreme precision. Tasks such as grasping a small drawer knob or pressing a specific microwave button are often prone to failure in traditional systems.

Robot gripper approaching a light switch, demonstrating the need for precise interaction with small objects.

Traditional open-loop "sense-plan-act" systems often fail due to small errors in navigation or calibration. Conversely, end-to-end imitation learning approaches require thousands of expensive real-world demonstrations and often struggle to generalize to novel environments or object variations. This paper addresses the "precision gap" by proposing a framework that reliably engages with tiny interaction sites without task-specific training.

Methodology

We introduce Servoing with Vision Models (SVM), a closed-loop framework that integrates classical visual servoing with modern Vision Foundation Models (VFMs). The pipeline consists of four primary stages:

  1. Target Specification: Targets are defined either semantically using an open-vocabulary detector (Detic) or via user-indicated point tracking (CoTracker).
  2. End-Effector Inpainting: To overcome the challenge of the robot's gripper occluding the target during approach, SVM utilizes a video inpainting model (ProPainter). This "paints out" the robot's arm, providing vision models with an unobstructed view for more accurate localization.
  3. Visual Servoing: The system continuously localizes the target in 3D and computes whole-body velocity commands to minimize the error between the gripper and the interaction site.
  4. Execution: Upon reaching a precise "pre-grasp" pose, the robot executes a simple motion primitive, such as closing the gripper or pushing forward.

Results and Discussion

The SVM framework was evaluated in 10 novel environments across 6 different buildings, involving 72 unique object instances. The system achieved a 71% success rate across all tasks in these novel environments, significantly outperforming the open-loop baseline (29% success rate).

Successful cabinet opening task after execution.

Key findings include:

  • The Role of Inpainting: Removing the inpainting module caused the success rate to drop from 71% to 62%, highlighting the importance of mitigating occlusion for off-the-shelf vision models.
  • Generalization vs. Training: SVM outperformed the Robot Utility Models (RUM) baseline—trained on over 1,700 demonstrations—by an absolute success rate of 50% on cabinet and drawer tasks. This suggests that leveraging the broad capabilities of foundation models can be more effective for geometric precision tasks than large-scale imitation learning.
  • Robustness: While semantic detectors struggled with partial views, point trackers like CoTracker maintained reliable locks on targets even as the gripper approached.

Conclusion

SVM demonstrates that a training-free, modular approach using vision foundation models can achieve high precision in mobile manipulation. While the current system faces challenges related to computational latency (0.1 Hz update rate) and depth sensor reliability on reflective surfaces, it provides a robust path forward for "in-the-wild" robotic interaction. Future work will explore the use of Large Language Models (LLMs) to automatically decompose high-level instructions into the necessary interaction sites and motion primitives.

Summary updated

Abstract

Many everyday mobile manipulation tasks require precise interaction with small objects, such as grasping a knob to open a cabinet or pressing a light switch. In this paper, we develop Servoing with Vision Models (SVM), a closed-loop training-free framework that enables a mobile manipulator to tackle such precise tasks involving the manipulation of small objects. SVM employs an RGB-D wrist camera and uses visual servoing for control. Our novelty lies in the use of state-of-the-art vision models to reliably compute 3D targets from the wrist image for diverse tasks and under occlusion due to the end-effector. To mitigate occlusion artifacts, we employ vision models to out-paint the end-effector thereby significantly enhancing target localization. We demonstrate that aided by out-painting methods, open-vocabulary object detectors can serve as a drop-in module to identify semantic targets (e.g. knobs) and point tracking methods can reliably track interaction sites indicated by user clicks. This training-free method obtains an 85% zero-shot success rate on manipulating unseen objects in novel environments in the real world, outperforming an open-loop control method and an imitation learning baseline trained on 1000+ demonstrations by an absolute success rate of 50%.
BibTeX
@article{gupta2026training,
  author = {Arjun Gupta and Rishik Sathua and Saurabh Gupta},
  title = {{A Training-Free Framework for Precise Mobile Manipulation of Small Everyday Objects}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2026},
  doi = {10.1109/lra.2026.3656784},
}
Figure