Task-Directed Exploration in Continuous POMDPs for Robotic Manipulation of Articulated Objects

Aidan Curtis, Leslie Pack Kaelbling, Siddarth Jain

TL;DR

Researchers have developed STRUG (Search with Task Relevant Uncertainty Guidance), a new online solver for robots operating under uncertainty. Unlike traditional methods that try to resolve all unknowns, STRUG identifies and prioritizes only the information necessary to complete a specific task, allowing robots to ignore "distractions" and successfully manipulate complex articulated objects like cabinets and drawers.

What Problem Does This Paper Address?

Robots often operate in "partially observable" environments where they aren't 100% sure about the state of the world—for example, the exact hinge location of a cabinet door or the contents of a drawer. The standard mathematical framework for this is the Partially Observable Markov Decision Process (POMDP).

While POMDP solvers have improved, they face two major hurdles:

  1. Long Horizons: Finding a sequence of actions that leads to a goal far in the future is computationally "doubly exponential."
  2. Task-Irrelevant Uncertainty: In the real world, there is infinite stuff to learn. A robot shouldn't waste time figuring out how every drawer in a kitchen opens if it only needs to put a cup in one specific cabinet. Most current "active learning" or "entropy-based" solvers try to reduce all uncertainty, which leads to inefficient behavior.

Methods

The authors introduce STRUG, which stands for Search with Task Relevant Uncertainty Guidance. The core innovation is a new metric for Task-Relevant Uncertainty (TRU).

  • Plan Compatibility Matrix: STRUG starts by sampling several possible versions of the world (particles). For each version, it uses a fast, "uncertainty-free" planner to find an ideal plan. It then tests every plan against every version of the world.
  • The "Aha" Metric: If a single plan works across many different versions of the world, the uncertainty between those versions is irrelevant. If different versions of the world require completely different plans to succeed, that uncertainty is task-relevant.
  • Augmented Reward: STRUG adds a "bonus" to the robot's reward function for taking actions that provide information specifically about these task-relevant differences.
  • Hierarchical Progressive Widening (HPW): To handle the infinite variety of continuous robotic movements (like exactly where to grab a handle), the authors developed HPW. This technique intelligently narrows down the search space by first choosing a high-level controller (e.g., "Open Drawer") and then refining the specific physical parameters.

Datasets

The researchers evaluated STRUG using:

  • Toy Benchmarks: Classic POMDP problems like the "Tiger" and "LightDark" tasks, modified to be more difficult.
  • Articulated Object Manipulation: They used the Sapien Partnet Mobility dataset for complex 3D models of furniture (shelves, drawers) and the YCB object set for items the robot had to manipulate.
  • Perception Pipeline: They used MaskRCNN (trained on simulated data) to detect object parts and Where2Act to generate heatmaps for potential interaction points.

Key Insights and Ideas

  • Uncertainty is not created equal: The most important takeaway is that robots should be "stubbornly goal-oriented" in their curiosity. Information gathering is only valuable if it has the potential to change the robot's next optimal action.
  • Dissimilar states can share plans: Even if two world models look different, they might be functionally identical for the task at hand. STRUG exploits this to skip unnecessary exploration.
  • Efficiency through Approximation: Calculating perfect task-relevant uncertainty is too slow for real-time use. The authors found that using a "cached" version of the plan compatibility matrix from the start of the task is a highly effective approximation.

Key Results

The authors compared STRUG against standard baselines like POMCPOW and "Entropy Regularized" solvers.

  • Complex Manipulation: In the "Open Drawer" and "Open Door" tasks, standard solvers almost always failed because they couldn't plan far enough ahead or got bogged down in irrelevant details. STRUG achieved significantly higher success rates (scoring 4.22 and 5.21 respectively, where baselines scored near 0 or 1).
  • Navigation: In the "LightDark 2D" task, where a robot must move toward light to find its position, STRUG successfully ignored "light walls" that didn't help it reach the goal, whereas entropy-based solvers wasted time visiting every light source.
  • Robustness: STRUG proved more efficient because it only interacted with the specific parts of an articulated object (like one specific drawer) needed for the goal.

Limitations and Future Work

  • Irreversible Actions: The current version of STRUG assumes that exploration actions don't "break" the environment. If a robot drops a glass vase to see if it's fragile, it can't undo that to reach the goal.
  • Static Latent Parameters: The math assumes the underlying "hidden" variables (like joint friction) don't change during the task.
  • Scaling Up: While successful in simulation, the authors aim to move this to even larger-scale environments with more controllers, objects, and complex temporal dependencies.

Summary updated

Abstract

Representing and reasoning about uncertainty is crucial for autonomous agents acting in partially observable environments with noisy sensors. Partially observable Markov decision processes (POMDPs) serve as a general framework for representing problems in which uncertainty is an important factor. Online sample-based POMDP methods have emerged as efficient approaches to solving large POMDPs and have been shown to extend to continuous domains. However, these solutions struggle to find long-horizon plans in problems with significant uncertainty. Exploration heuristics can help guide planning, but many real-world settings contain significant task-irrelevant uncertainty that might distract from the task objective. In this paper, we propose STRUG, an online POMDP solver capable of handling domains that require long-horizon planning with significant task-relevant and task-irrelevant uncertainty. We demonstrate our solution on several temporally extended versions of toy POMDP problems as well as robotic manipulation of articulated objects using a neural perception frontend to construct a distribution of possible models. Our results show that STRUG outperforms the current sample-based online POMDP solvers on several tasks.
BibTeX
@inproceedings{curtis2023task,
  author = {Aidan Curtis and Leslie Pack Kaelbling and Siddarth Jain},
  title = {{Task-Directed Exploration in Continuous POMDPs for Robotic Manipulation of Articulated Objects}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=YjUUcoCvTW_},
}

Loading related content...

Figure