TL;DR
This paper presents an improved assistive robotic system that can autonomously pick and place novel objects in cluttered, open-world environments. The system leverages a laser pointer for user interaction and a novel augmented reality interface for feedback, achieving significantly faster and more reliable pick-and-place operations compared to previous work.
What Problem Does This Paper Address?
Assistive robotics holds immense potential for supporting individuals in their daily lives, particularly with the growing elderly population and the millions living with disabilities that affect motor functions. A crucial capability for such robots is the ability to autonomously pick and place a wide variety of novel objects in unstructured environments. However, current assistive robots often lack this flexibility, requiring pre-programmed objects or highly controlled settings. Furthermore, existing systems can be difficult to learn and use. This paper aims to bridge this gap by developing a more capable and user-friendly assistive robotic system.
Methods
The core of the system is a Universal Robotics UR5 robotic arm mounted on a mobility scooter. The system employs a multi-sensor approach with five depth sensors to create a comprehensive point cloud of the workspace, avoiding the need for metric SLAM.
Key methodological contributions include:
- User Interaction via Laser Pointer and DSAR: Users select objects and placement locations using a dual laser pointer. The system provides feedback through a Dynamic Spatial Augmented Reality (DSAR) interface, projecting light directly onto the real world to indicate grasp intent and target locations.
- Grasp Detection and Selection: The system utilizes the Grasp Pose Detection (GPD) library for detecting 6-DOF grasp poses from point cloud data. It incorporates a novel strategy for manually selecting grasps, which significantly reduces errors in cluttered environments. An automatic grasp selection mechanism prioritizes grasps based on an objective function considering factors like grasp width, reachability, approach vector, proximity to the object, and distance to the laser pointer.
- Pick-and-Place Functionality: The system extends previous pick-and-drop capabilities to full pick-and-place. For placing, it adjusts for the object's height based on the grasp pose and uses the laser pointer to define the placement location.
- Stiff Manipulator Hardware: Unlike some prior work that uses compliant manipulators for safety, this system opts for stiff hardware, which can lead to more precise movements.
Datasets
The paper does not explicitly mention training datasets. For evaluation, the authors used a set of 30 objects similar to those used in prior work [5] for a fair comparison. They also conducted experiments in a real-world kitchen environment with various surfaces and object placements.
Key Insights and Ideas
- DSAR for Intuitive Feedback: Projecting visual cues directly into the real world (DSAR) offers a more intuitive and less intrusive way to communicate robot intent to the user compared to traditional screen-based interfaces.
- Multi-Sensor Fusion for Robust Perception: Using multiple depth sensors strategically placed provides better workspace coverage and reduces occlusion, leading to more reliable perception for grasp detection.
- Combining Automatic and Manual Grasp Selection: The system offers both automatic and manual grasp selection, acknowledging that while automatic methods are efficient, manual selection is crucial for handling complex scenarios or user preferences, especially in preventing "wrong object" failures.
- Stiff Manipulators for Precision: The choice of stiff manipulator hardware, despite potential safety concerns, appears to contribute to improved performance in terms of grasp success and execution.
Key Results
- Grasping in Isolation: Achieved a grasp success rate of 93.8% and a detection success rate of 92.3%, representing a 4.2% and 4.5% improvement, respectively, over prior work [5]. The average grasp time was reduced to 51.7s.
- Grasping In-Situ: Demonstrated a task success rate of 96% in a more challenging kitchen environment with objects placed on shelves and tables. This showed a 10% higher task success rate and a 24% lower grasp execution failure rate compared to [5].
- Pick-and-Place In-Situ: Achieved a 96.4% task success rate for pick-and-place operations. The average time for a pick-and-place task was 130s (excluding driving).
- Manual vs. Automatic Grasp Selection: Manual grasp selection resulted in a 20% lower "wrong object" failure rate, while automatic grasp selection showed a 6% lower grasp execution failure rate.
Limitations and Future Work
The authors acknowledge several limitations and suggest future research directions:
- Motion Planning Failures: The system occasionally struggles to find a feasible motion plan. This could be addressed by exploring alternative motion planning algorithms.
- Laser Pointer Detection: Laser pointer detection can sometimes fail, especially in environments with interfering light sources. Future work could involve using RGB-D sensors and exploiting RGB information to improve detection.
- Runtime Optimization: Further reducing runtime could be achieved by using stereo cameras to avoid infrared interference, optimizing grasp pose detection through parallel computation, and improving sensor switching strategies.
- Pick-and-Place Simplicity: The current pick-and-place functionality primarily focuses on target position and largely ignores object orientation. Future work aims to allow users to specify target orientations and enable object reorientation.
- User Interface Evolution: The authors envision removing the screen entirely, using the projector to display more information about planned grasps or workspace, and developing a truly "real-world" interface.
- User Studies: Conducting user studies with diverse user groups is planned to adapt the interface for different abilities and demographics, leading to a more robust and universally applicable system.
Summary updated