Mobile Robot Manipulation Using Pure Object Detection

Brent GriffinWACV 2023

TL;DR

This paper introduces ClickBot, a mobile robot that learns to manipulate objects using only object detection, eliminating the need for 3D models or prior knowledge of object shapes. It achieves this through a novel Task-Focused Few-Shot Object Detection (TFOD) framework, enabling the robot to collect its own data and learn new tasks with minimal human annotation.

What Problem Does This Paper Address?

Traditional mobile robot manipulation relies heavily on pre-programmed models, extensive datasets, or complex sensor setups (like RGBD cameras). This makes it difficult for robots to adapt to new objects, changing environments, or tasks they haven't been explicitly trained for. Off-the-shelf object detectors, while powerful, often fail when deployed in novel scenarios. The paper addresses this by proposing a system that leverages the robot's own interactions with the world to learn manipulation skills, specifically by using object detection as the primary sensory input and learning mechanism.

Methods

The core of the paper is the ClickBot system, which integrates two main innovations:

  • Detection-Based Manipulation: ClickBot performs manipulation tasks (finding, servoing, depth estimation, grasping) using only 2D bounding boxes from an object detector.
  • Visual Servo Control (VS): A novel update formulation for estimating the feature Jacobian (Ls) is introduced, allowing the robot to learn visual control faster and more consistently than prior methods. This formulation uses the robot's own motion and observed changes in detection error to update its understanding of how camera movements affect object detection.
  • Motion-Based Depth Estimation: The robot actively moves to estimate depth, improving accuracy as it gets closer to the object.
  • Active Grasp Planning: The robot uses detection bounding boxes to plan antipodal grasps, adapting to object shapes without explicit 3D models.
  • Task-Focused Few-Shot Object Detection (TFOD): This framework allows the robot to learn new objects and adapt its detection model to specific tasks.
  • Interactive Learning: When the robot encounters a failure (e.g., cannot detect an object for a task), it saves the relevant image. A human then annotates this image with a bounding box (taking about 7 seconds per object).
  • Automatic Data Selection: The robot automatically identifies which tasks require more annotation, focusing human effort on the most critical areas for improvement. This significantly reduces the overall annotation cost compared to traditional methods.

Datasets

  • YCB Dataset [6]: Used for evaluating VS, depth estimation, and pick-and-place tasks.
  • MS-COCO [36]: The baseline object detection model (Faster R-CNN) was trained on this dataset.
  • TFOD Benchmark (New): The authors introduce a new benchmark for evaluating TFOD. It's configurable for various few-shot settings (k=1, 2, 4 examples) and uses YCB objects in challenging, cluttered environments. This benchmark aims to guide future research in task-specific object detection for robotics.

Key Insights and Ideas

  • Pure Detection for Manipulation is Feasible: The paper demonstrates that robots can learn complex manipulation tasks like grasping and placing objects using only 2D object detection, challenging the notion that 3D models or depth sensors are always necessary.
  • Robot-Centric Data Collection is Powerful: By allowing the robot to collect its own data when it encounters failures, the system can efficiently learn to overcome specific challenges and adapt to new scenarios.
  • Few-Shot Learning for Robotics is Task-Specific: TFOD highlights that the "few-shot" examples needed are highly dependent on the specific task and environment, and the robot can intelligently guide this annotation process.
  • Annotation Efficiency: The TFOD approach drastically reduces annotation time (e.g., 26 seconds per object vs. 540 seconds for segmentation masks), making robot learning more practical.

Key Results

  • Visual Servo Control: ClickBot learns VS in an average of 13.3 seconds with 13 updates, requiring 30-60% fewer updates and showing 15-35% less parameter variation than prior methods.
  • VOSVS Benchmark: ClickBot achieved a perfect VS score and improved depth estimation success rate from 42% to 67%.
  • Pick-and-Place in Clutter: Achieved an 88% cluttered grasp and pick-and-place rate using only four few-shot examples, outperforming methods that use RGBD cameras.
  • Dynamic Placement: Demonstrated manipulation with moving placement locations at a rate of 124.6 picks-per-hour.
  • Annotation Reduction: TFOD achieved a 95% reduction in annotation time compared to segmentation-based methods.

Limitations and Future Work

  • Limitations:
    • The current system relies on human annotation for TFOD.
  • While robust, the system still has room for improvement in depth estimation for certain object categories (Tool and Kitchen sets).
  • The paper focuses on single-object manipulation primarily, though it can handle clutter.
  • Future Work:
    • Expand to more complex manipulation tasks, such as those involving multiple cluttered rooms.
  • Further develop innovations in object detection to improve performance, especially for small objects and low-shot scenarios.
  • Release the TFOD Benchmark and its associated data to foster further research in task-focused object detection for robotics.
  • Explore releasing additional datasets and application areas for the benchmark.

Summary updated

BibTeX
@inproceedings{griffin2023mobile,
  author = {Brent Griffin},
  title = {{Mobile Robot Manipulation Using Pure Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure