Memory-centered and Affordance-based Framework for Mobile Manipulation

Christoph Pohl, Fabian Reister, Fabian Peller-Konrad, Tamim Asfour

Abstract

Developing mobile manipulation skills that are transferable across different robotic platforms and environments remains a significant challenge in robotics. This paper introduces MAkEable, a framework designed to enable "triple-mode transfer": the ability to reuse skills across varying tasks, robot hardware, and environments. By utilizing an affordance-based task representation and a centralized memory system, MAkEable allows robots to share and adapt complex manipulation skills—such as grasping, pouring, or opening drawers—without requiring platform-specific reprogramming.

Introduction

In contemporary robotics, manipulation skills are frequently "siloed," meaning a skill developed for a specific robot in a controlled environment rarely translates to a different hardware platform or a new setting. The authors identify three critical modes of transfer that are typically addressed in isolation: across tasks (e.g., reusing a grasp for different objects), across robots (e.g., moving a skill from a high-end humanoid to a mobile arm), and across environments (e.g., adapting a skill from a lab to a home kitchen).

The MAkEable framework addresses the challenge of transferring manipulation skills across three primary dimensions: Robot, Task, and Environment.

The primary hurdle is establishing a "universal language" for tasks—one that is abstract enough to be robot-agnostic yet sufficiently detailed for real-world execution. MAkEable addresses this by decoupling the high-level intent of an action from its low-level physical execution.

Methodology: The MAkEable Framework

The core of MAkEable is its Interpretable Data Format (IDF) and a five-step execution pipeline integrated into the ArmarX cognitive architecture.

The Five-Step Execution Pipeline

  1. Discovery (D): Identifying potential interaction points (e.g., using Segment Anything (SAM) for zero-shot object segmentation).
  2. Parameterization (P): Converting abstract hypotheses into specific plans (e.g., determining the optimal base position for a specific robot).
  3. Validation (V): Assessing feasibility through reachability analysis and collision checking.
  4. Selection (S): Choosing the optimal action based on performance criteria like execution height.
  5. Execution (E): Implementing the low-level control loops.
The MAkEable architecture integrates high-level skills with low-level control through the ArmarX Memory System, facilitating the transfer of knowledge between different robotic platforms.

Memory-Centered Architecture

Central to the framework is a Long-Term Memory (LTM) system. When a robot learns a skill, it is stored in a generalized format. This allows other robots to query the memory and perform the same task asynchronously, benefiting from demonstrations performed by different machines in the past.

Key Insights

  • ActionHypothesis vs. ExecutableAction: The framework distinguishes between a robot-agnostic ActionHypothesis (e.g., "pull this handle at this angle") and a robot-specific ExecutableAction (e.g., specific joint angles for a particular arm). This separation is the key to seamless transfer.
  • Strategy-Based Design: Each step of the pipeline is modular. Researchers can swap out individual strategies—such as replacing a geometric discovery method with a neural network—without redesigning the entire manipulation stack.

Experimental Results

The framework's versatility was validated through several real-world scenarios:

  • Table Clearing: Different humanoid models (ARMAR-6 and ARMAR-DE) successfully cleared tables, placing objects in semantically appropriate locations like sinks or cupboards.
  • Skill Transfer: A drawer-opening skill taught to one robot via kinesthetic demonstration was successfully retrieved and executed by a different robot using its opposite arm.
  • Cross-Platform Pouring: A pouring skill derived from human data was successfully transferred to "Omni-Frankie," a non-humanoid mobile manipulator, demonstrating the framework's independence from specific robot morphologies.

Discussion and Future Work

While MAkEable provides a robust foundation for skill transfer, current limitations include a reliance on pre-planned trajectories rather than reactive feedback. Future work will focus on incorporating real-time tactile and visual feedback to improve robustness against slips and moving objects. Additionally, the authors aim to develop standardized evaluation metrics to compare MAkEable against other emerging Open Vocabulary Mobile Manipulation (OVMM) frameworks.

Summary updated

BibTeX
@inproceedings{pohl2024memory,
  author = {Christoph Pohl and Fabian Reister and Fabian Peller-Konrad and Tamim Asfour},
  title = {{Memory-centered and Affordance-based Framework for Mobile Manipulation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10801986},
}
Figure