Semantic-based Loco-Manipulation for Human-Robot Collaboration in Industrial Environments

Federico Rollo, Gennaro Raiola, Nikolaos G. Tsagarakis, Marco Roveri, Enrico Mingo Hoffman, Arash Ajoudani

Abstract

This paper introduces a semantic loco-manipulation framework designed for mobile robots, specifically quadrupeds equipped with robotic arms, to perform collaborative "fetch and deliver" tasks. By integrating high-level semantic mapping with real-time instance segmentation and Behavior Trees (BT), the system enables autonomous navigation, object identification, and precise manipulation in dynamic industrial settings.

Introduction

In modern "smart" industrial environments, the integration of autonomous robots aims to alleviate worker strain by delegating auxiliary tasks. A primary challenge is the "fetch" task, where a worker requires a specific tool while their attention is occupied. To execute this, a robot must bridge the gap between high-level semantics (understanding what an object is) and precise geometry (locating and reaching it in 3D space). Most current solutions treat navigation and manipulation as isolated problems; this research proposes a unified loco-manipulation framework within a human-centric workflow.

Methodology

The proposed architecture is orchestrated by a Behavior Tree (BT) that manages six primary actions to ensure robust execution and safety:

  1. Navigate to Object: The robot utilizes a pre-existing semantic map to reach the general vicinity of the target.
  2. Approach Object: Using a YOLOv8 instance segmentation network and RGB-D data, the robot identifies the object's centroid and uses linear proportional controllers to position itself within its optimal manipulability space.
  3. Pose Estimation: The system computes the precise homogeneous transformation required for a successful grasp.
  4. Pick Object: Leveraging the Whole-body Locomotion Framework (WoLF), the robot executes optimization-based inverse dynamics to move its arm and gripper to the target pose.
  5. Bring to User: The robot navigates back to the human's last known location.
  6. Release Object: The final hand-off to the human worker.
Figure 1: Sequence of the semantic loco-manipulation pipeline, illustrating navigation, object identification, grasping, and delivery to a human user in a simulated environment.

A critical safety feature is the "Abort" state, which halts all goals if any action fails, ensuring the robot enters a safe idle state rather than performing unpredictable movements.

Experimental Framework

The framework was validated using the WoLF Simulation Environment, a custom world populated with dynamic human agents and various artifacts. The vision system utilized pre-trained YOLOv8 weights for real-time instance segmentation. This setup allowed for testing the pipeline's robustness across various random configurations of objects and people.

Results and Discussion

The study successfully integrated several complex software stacks, including ROS Navigation, YOLOv8 for vision, and WoLF for whole-body control. Key findings include:

  • Semantic-Geometric Coupling: The transition from a rough semantic goal to a precise geometric action is achieved through a visual servoing phase that compensates for global navigation map imprecision.
  • Flexibility of Behavior Trees: Unlike traditional state machines, BTs provided a human-readable and flexible method for managing complex missions, allowing for easy insertion of safety behaviors.
  • Reliability: The pipeline consistently completed tasks across varying world configurations, demonstrating qualitative success in navigating cluttered environments to deliver specific items to users.

Conclusion and Future Work

While the simulation results are promising, several areas for future development remain. The current system assumes simplified grasping orientations, which the authors plan to generalize for complex geometries. Additionally, future work will address the "sim-to-real" gap, testing the framework against real-world sensor noise and lighting conditions. Enhancing the robot's "self-healing" capabilities—such as autonomously re-attempting a grasp if an object is dropped—is also a priority for improving mission success rates.

Summary updated

BibTeX
@inproceedings{rollo2024semantic,
  author = {Federico Rollo and Gennaro Raiola and Nikolaos G. Tsagarakis and Marco Roveri and Enrico Mingo Hoffman and Arash Ajoudani},
  title = {{Semantic-based Loco-Manipulation for Human-Robot Collaboration in Industrial Environments}},
  booktitle = {Erf},
  year = {2024},
  doi = {10.1007/978-3-031-76428-8\_11},
}
Figure