Abstract
This paper introduces a semantic loco-manipulation framework designed for mobile robots, specifically quadrupeds equipped with robotic arms, to perform collaborative "fetch and deliver" tasks. By integrating high-level semantic mapping with real-time instance segmentation and Behavior Trees (BT), the system enables autonomous navigation, object identification, and precise manipulation in dynamic industrial settings.
Introduction
In modern "smart" industrial environments, the integration of autonomous robots aims to alleviate worker strain by delegating auxiliary tasks. A primary challenge is the "fetch" task, where a worker requires a specific tool while their attention is occupied. To execute this, a robot must bridge the gap between high-level semantics (understanding what an object is) and precise geometry (locating and reaching it in 3D space). Most current solutions treat navigation and manipulation as isolated problems; this research proposes a unified loco-manipulation framework within a human-centric workflow.
Methodology
The proposed architecture is orchestrated by a Behavior Tree (BT) that manages six primary actions to ensure robust execution and safety:
- Navigate to Object: The robot utilizes a pre-existing semantic map to reach the general vicinity of the target.
- Approach Object: Using a YOLOv8 instance segmentation network and RGB-D data, the robot identifies the object's centroid and uses linear proportional controllers to position itself within its optimal manipulability space.
- Pose Estimation: The system computes the precise homogeneous transformation required for a successful grasp.
- Pick Object: Leveraging the Whole-body Locomotion Framework (WoLF), the robot executes optimization-based inverse dynamics to move its arm and gripper to the target pose.
- Bring to User: The robot navigates back to the human's last known location.
- Release Object: The final hand-off to the human worker.
A critical safety feature is the "Abort" state, which halts all goals if any action fails, ensuring the robot enters a safe idle state rather than performing unpredictable movements.
Experimental Framework
The framework was validated using the WoLF Simulation Environment, a custom world populated with dynamic human agents and various artifacts. The vision system utilized pre-trained YOLOv8 weights for real-time instance segmentation. This setup allowed for testing the pipeline's robustness across various random configurations of objects and people.
Results and Discussion
The study successfully integrated several complex software stacks, including ROS Navigation, YOLOv8 for vision, and WoLF for whole-body control. Key findings include:
- Semantic-Geometric Coupling: The transition from a rough semantic goal to a precise geometric action is achieved through a visual servoing phase that compensates for global navigation map imprecision.
- Flexibility of Behavior Trees: Unlike traditional state machines, BTs provided a human-readable and flexible method for managing complex missions, allowing for easy insertion of safety behaviors.
- Reliability: The pipeline consistently completed tasks across varying world configurations, demonstrating qualitative success in navigating cluttered environments to deliver specific items to users.
Conclusion and Future Work
While the simulation results are promising, several areas for future development remain. The current system assumes simplified grasping orientations, which the authors plan to generalize for complex geometries. Additionally, future work will address the "sim-to-real" gap, testing the framework against real-world sensor noise and lighting conditions. Enhancing the robot's "self-healing" capabilities—such as autonomously re-attempting a grasp if an object is dropped—is also a priority for improving mission success rates.
Summary updated