Abstract
Representing and reasoning about uncertainty is crucial for autonomous agents acting in partially observable environments with noisy sensors. Partially observable Markov decision processes (POMDPs) serve as a general framework for representing problems in which uncertainty is an important factor. Online sample-based POMDP methods have emerged as efficient approaches to solving large POMDPs and have been shown to extend to continuous domains. However, these solutions struggle to find long-horizon plans in problems with significant uncertainty. Exploration heuristics can help guide planning, but many real-world settings contain significant task-irrelevant uncertainty that might distract from the task objective. In this paper, we propose STRUG, an online POMDP solver capable of handling domains that require long-horizon planning with significant task-relevant and task-irrelevant uncertainty. We demonstrate our solution on several temporally extended versions of toy POMDP problems as well as robotic manipulation of articulated objects using a neural perception frontend to construct a distribution of possible models. Our results show that STRUG outperforms the current sample-based online POMDP solvers on several tasks.
Introduction
Robots operating in the real world face significant challenges due to partial observability. Whether it is the exact hinge location of a cabinet door or the contents of a drawer, uncertainty is pervasive. The Partially Observable Markov Decision Process (POMDP) provides a robust mathematical framework for these scenarios. However, traditional online POMDP solvers struggle with two main issues: the computational complexity of long-horizon planning and the distraction of task-irrelevant uncertainty. In many environments, there is an infinite amount of information a robot could learn, but most of it is irrelevant to the specific task at hand. Current "active learning" or entropy-based solvers often fail by attempting to reduce all uncertainty, leading to inefficient and wandering behaviors.
Methodology: STRUG
To address these challenges, we introduce STRUG (Search with Task Relevant Uncertainty Guidance). The core innovation is a metric for Task-Relevant Uncertainty (TRU) that allows a robot to prioritize information gathering only when it is necessary for task completion.
Plan Compatibility Matrix
STRUG operates by sampling multiple possible versions of the world (particles). For each version, it utilizes a fast, uncertainty-free planner to generate an ideal plan. The system then evaluates every plan against every sampled world model. If a single plan succeeds across various models, the uncertainty between those models is deemed irrelevant. Conversely, if different models require distinct plans to succeed, that uncertainty is task-relevant.
Task-Relevant Uncertainty Guidance
STRUG augments the robot's reward function with a bonus for actions that resolve task-relevant uncertainty. This ensures the robot only explores when the information gained has the potential to change its optimal next action. To handle the continuous action spaces inherent in robotic manipulation, we also developed Hierarchical Progressive Widening (HPW). This technique narrows the search space by first selecting a high-level controller (e.g., "Open Drawer") and then refining specific physical parameters.
Experimental Evaluation
We evaluated STRUG across toy benchmarks and complex robotic manipulation tasks. The manipulation tasks utilized the Sapien Partnet Mobility dataset for furniture models and the YCB object set for items. A perception pipeline involving MaskRCNN and Where2Act was used to detect object parts and generate interaction heatmaps.
In the "LightDark 2D" navigation task, a robot must move toward light sources to localize itself before reaching a goal. This task was modified to include "light walls"—sources of information that do not help in reaching the goal.
Key Results
STRUG significantly outperformed standard baselines like POMCPOW and entropy-regularized solvers. In complex manipulation tasks such as "Open Drawer" and "Open Door," standard solvers frequently failed due to the long planning horizons or distraction by irrelevant details. STRUG achieved success rates of 4.22 and 5.21 respectively, while baselines typically scored near 0 or 1.
As shown in the navigation results, entropy-regularized solvers wasted time visiting every light source, including the irrelevant light wall. STRUG, however, correctly identified that the light wall did not provide information necessary for reaching the goal and bypassed it.
Conclusion
The primary insight of this work is that uncertainty is not created equal. By being "stubbornly goal-oriented" in its curiosity, STRUG allows robots to ignore functional distractions and focus on the information required for task success. Future work will explore handling irreversible actions and scaling to even larger environments with more complex temporal dependencies.
Summary updated