A POMDP-based hierarchical planning framework for manipulation under pose uncertainty

Muhammad Suhail Saleem, Rishi Veerapaneni, Maxim LikhachevarXiv 2024 · DOI · arXiv

Abstract

This paper presents a hierarchical planning framework designed to enable robots to localize and manipulate objects using only binary contact signals when visual information is unavailable. By transitioning from a coarse volumetric belief representation to a fine-grained particle-based model, the system effectively addresses the computational complexity of Partially Observable Markov Decision Processes (POMDPs). The framework achieves a 93% success rate on high-precision tasks, such as plug insertion, even under significant initial pose uncertainty.

Introduction

In unstructured domestic or industrial environments, robots frequently encounter situations where vision is unreliable due to poor lighting, occlusions, or self-shadowing. In these scenarios, robots must rely on tactile feedback to "feel" their way through a task. Mathematically, this is modeled as a POMDP, where the robot must balance actions that gather information with actions that progress toward a goal. However, standard approaches like particle filters struggle with the "curse of dimensionality" when uncertainty spans large volumes, making real-time planning computationally prohibitive.

Methodology

The proposed framework utilizes a two-phase hierarchical approach to maintain tractability while ensuring high precision.

1. Volumetric Representation

In the initial phase, uncertainty is represented as a 3D occupancy grid. The robot performs "no-contact" observations: as the arm moves through space without hitting anything, it "carves away" the volume where the object cannot be. If contact occurs, the belief is updated to reflect that the object must be within a maximum distance ($D_{max}$) of the contact point. This abstraction allows for rapid reduction of the search space without the heavy computation required for mesh-to-mesh collision checks.

Volumetric carving using no-contact observations to reduce uncertainty.

2. Particle Representation

Once the volumetric uncertainty is sufficiently reduced, the system switches to a particle-based representation. This "zoomed-in" view provides the geometric fidelity necessary for final manipulation steps, such as aligning a plug with a socket.

The Planner: Modified RTDP-Bel

To meet a strict 1-second planning budget per step, the authors modified the Real-Time Dynamic Programming (RTDP) algorithm with two key features:

  • Outcome Pruning: The planner focuses on the most likely outcomes while maintaining value estimates for less probable branches.
  • Dual Heuristics: The system uses a schedule of heuristics, starting with admissible (optimistic) heuristics to encourage exploration and switching to inadmissible (greedy) heuristics to ensure convergence within the time limit.

Experimental Setup

The framework was validated using a UR10e robotic arm in both simulation and real-world environments. The primary task involved inserting a plug into a port under various conditions:

  • Shelf Scenario: The port is hidden inside a shelf, introducing uncertainty in $y, z,$ and $roll$.
  • Movable Base Scenario: The port is on a movable stand, creating uncertainty in $x, y, z,$ and $yaw$.
Experimental setups including the shelf scenario and plug insertion task.

Results and Discussion

The hierarchical framework demonstrated high reliability and efficiency:

  • Success Rate: The system achieved a 93% success rate in real-world trials (55/60 successful runs).
  • Planning Quality: Compared to greedy baselines that prioritize immediate information gain, this framework improved solution quality by over 50%.
  • Robustness: The system successfully resolved positional uncertainties of up to 50 cm and angular uncertainties covering a full $2\pi$ range.
  • Efficiency: Average execution times ranged between 33 and 42 seconds, making it viable for practical domestic applications.

Conclusion

While highly effective, the system faces limitations regarding discretization errors; if the true object pose falls between sampled points, the filter may fail. Additionally, the current model assumes a static environment where the target does not move upon contact. Future work will explore dynamic resolution for volumetric grids and more complex object geometries to further enhance the robot's ability to operate in the dark.

Summary updated

Abstract

Robots often face challenges in domestic environments where visual feedback is ineffective, such as retrieving objects obstructed by occlusions or finding a light switch in the dark. In these cases, utilizing contacts to localize the target object can be effective. We propose an online planning framework using binary contact signals for manipulation tasks with pose uncertainty, formulated as a Partially Observable Markov Decision Process (POMDP). Naively representing the belief as a particle set makes planning infeasible due to the large uncertainties in domestic settings, as identifying the best sequence of actions requires rolling out thousands of actions across millions of particles, taking significant compute time. To address this, we propose a hierarchical belief representation. Initially, we represent the uncertainty coarsely in a 3D volumetric space. Policies that refine uncertainty in this space are computed and executed, and once uncertainty is sufficiently reduced, the problem is translated back into the particle space for further refinement before task completion. We utilize a closed-loop planning and execution framework with a heuristic-search-based anytime solver that computes partial policies within a limited time budget. The performance of the framework is demonstrated both in real world and in simulation on the high-precision task of inserting a plug into a port using a UR10e manipulator, resolving positional uncertainties up to 50 centimeters and angular uncertainties close to $2π$. Experimental results highlight the framework's effectiveness, achieving a 93\% success rate in the real world and over 50\% improvement in solution quality compared to greedy baselines, significantly accelerating planning and enabling real-time solutions for complex problems.
BibTeX
@misc{saleem2024pomdp,
  author = {Muhammad Suhail Saleem and Rishi Veerapaneni and Maxim Likhachev},
  title = {{A POMDP-based hierarchical planning framework for manipulation under pose uncertainty}},
  year = {2024},
  eprint = {2409.18775},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.18775},
}
Figure