Planning and Reinforcement Learning for General-Purpose Service Robots

Yuqian JiangIJCAI 2021 · DOI

TL;DR

This research tackles the challenge of creating general-purpose service robots that can adapt to real-world complexities. It proposes a novel integration of symbolic planning and reinforcement learning to enable robots to reason about high-level tasks, adapt to dynamic environments, and interact more robustly with humans.

What Problem Does This Paper Address?

The dream of general-purpose service robots—robots that can seamlessly operate in homes and public spaces, performing diverse tasks like cooking, cleaning, and delivery—faces a significant hurdle. While deep learning has advanced low-level robot skills, current robots struggle with high-level task control in unpredictable, real-world environments. This gap arises because learned skills alone are insufficient for tasks requiring complex reasoning, such as understanding ambiguous human requests, planning sequences of actions, and adapting to unforeseen circumstances. Traditional symbolic planning methods, while offering guarantees, are often brittle in dynamic, uncertain settings. This paper aims to bridge this gap by combining the strengths of symbolic planning (for structured reasoning) and reinforcement learning (for adaptation and learning from experience).

Methods

The core of this research lies in integrating symbolic planning and reinforcement learning. The authors propose several key contributions:

  • Open-World Reasoning and Planning: A method to enable robots to reason about hypothetical objects mentioned by human users, allowing them to plan even when their knowledge base is incomplete.
  • Multi-Robot Planning with Conflicts and Synergies: A framework (Iterative Inter-Dependent Planning - IIDP) to address situations where multiple robots have conflicting resource needs or can benefit from each other's actions, enabling coordinated planning.
  • Task-Motion Planning with Cost Learning (TMP-CL): A framework that iteratively plans and updates action costs, learning from motion planner feedback to generate low-cost, feasible task-motion plans. This framework also learns actual action costs during execution to improve future plans.
  • Reinforcement Learning for Continual Tasks: Development of methods for average-reward reinforcement learning, which is more suitable for long-term robot deployments than traditional episodic RL. This includes applying RL to continual area sweeping for environment knowledge updates.
  • Temporal-Logic-Based Reward Shaping: A novel approach to automatically translate temporal-logic formulas into reward shaping functions, allowing for efficient learning of average-reward RL tasks by incorporating domain knowledge without manual crafting.
  • Future Work: RL for Task-Motion Planning: The authors plan to integrate model-free RL for action selection within the TMP-CL framework, leveraging the initial plans and fast convergence of TMP-CL for improved robustness and adaptation to unexpected outcomes.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Bridging the Gap: The fundamental insight is that neither pure symbolic planning nor pure reinforcement learning is sufficient for general-purpose service robots. A hybrid approach is necessary to combine structured reasoning with adaptive learning.
  • Handling Uncertainty: Real-world service robots will inevitably encounter incomplete information and dynamic environments. The proposed methods for open-world reasoning and learning action costs directly address this challenge.
  • Long-Term Robot Autonomy: The focus on average-reward RL is crucial for robots that operate continuously, requiring them to optimize for sustained performance rather than just single episodes.
  • Leveraging Domain Knowledge: Reward shaping, particularly when automated via temporal logic, offers a powerful way to inject prior knowledge into RL, significantly speeding up learning and improving performance.
  • Synergy in Multi-Robot Systems: Recognizing and exploiting conflicts and synergies among multiple robots is key to efficient and collaborative operation.

Key Results

The paper presents contributions that have been published in various conferences and journals. Specific quantitative results are not detailed in this abstract but are implied to be positive and significant for the respective contributions:

  • Open-World Reasoning: A method introduced that reasons about hypothetical objects for planning in the open world.
  • Multi-Robot Planning: Development of the IIDP framework for multi-robot planning with conflicts and synergies.
  • Task-Motion Planning: The TMP-CL framework generates low-cost, feasible task-motion plans and learns actual action costs.
  • Continual Area Sweeping: A deep RL algorithm for non-uniform environment surveying.
  • Reward Shaping: A framework for temporal-logic-based reward shaping for average-reward RL.

Limitations and Future Work

  • Integration and Evaluation: The authors acknowledge that the full integration and evaluation of these contributions on real robots are future steps.
  • Unexpected Action Outcomes: The current TMP-CL framework does not fully account for unexpected action outcomes, which is a key area for future work involving model-free RL.
  • Manual Action Definitions: The symbolic planner's action definitions are manually specified, limiting their adaptability. Future work aims to address this through RL integration.
  • Real-World Deployment: While the research aims for general-purpose service robots, the abstract focuses on the foundational contributions rather than a fully deployed system. The ultimate goal is to evaluate these integrated contributions on real robots for tasks like delivering objects and tidying.

Summary updated

BibTeX
@inproceedings{jiang2021planning,
  author = {Yuqian Jiang},
  title = {{Planning and Reinforcement Learning for General-Purpose Service Robots}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/679},
}
Figure