TL;DR
This paper introduces HiTUT, a novel hierarchical task learning framework that decomposes complex language-guided tasks into sub-goal planning, scene navigation, and object manipulation. By unifying these components with transformers and incorporating self-monitoring and backtracking, HiTUT achieves state-of-the-art performance on the ALFRED benchmark, particularly excelling in generalization to unseen environments.
What Problem Does This Paper Address?
The paper tackles the significant challenge of teaching robots to perform complex tasks based on natural language instructions. While current methods often use end-to-end approaches, they struggle with generalization, achieving less than 10% success rate on the ALFRED benchmark in unseen environments, a stark contrast to human performance (over 90%). This lack of generalization and the "black box" nature of end-to-end models make it difficult to understand why they succeed or fail. The authors aim to improve both performance and interpretability by introducing a hierarchical approach.
Methods
The core of the proposed method, HiTUT (Hierarchical Tasks via Unified Transformers), is a departure from end-to-end architectures. It decomposes task learning into three distinct but interconnected sub-problems:
- Sub-Goal Planning: Predicts a sequence of intermediate sub-goals (e.g., "go to the sink") to achieve the high-level goal. This is done "on the fly" after each sub-goal is executed, making it more adaptive than fixed plans.
- Scene Navigation: Handles the low-level actions (e.g., "move forward," "turn left") required to reach a target location specified by a sub-goal.
- Object Manipulation: Manages actions like "pick up" or "clean" and crucially, grounds these actions to specific objects in the environment by predicting segmentation masks.
These three sub-problems are addressed using a unified transformer architecture. Instead of recurrent state transitions, HiTUT feeds prediction history as input to subsequent predictions. It also leverages pre-trained object detectors (Mask R-CNN) for visual grounding and uses predicate embeddings to bridge symbolic representations with word embeddings. A key innovation is the self-monitoring and backtracking mechanism, allowing the agent to detect failures in sub-goals and attempt alternative plans, significantly improving robustness.
Datasets
The primary dataset used is ALFRED (A benchmark for interpreting grounded instructions for everyday tasks). The authors follow the standard train/validation/test splits, further categorizing them into "seen" and "unseen" environments to evaluate generalization. They also mention pre-training the object detector on MSCOCO.
Key Insights and Ideas
- Hierarchical Structure is Key for Generalization: Decomposing tasks into explicit sub-goals significantly boosts the agent's ability to generalize to novel environments, outperforming end-to-end models by over 160%.
- Navigation is the Bottleneck: The analysis reveals that navigation actions are considerably harder for the agent to learn than sub-goal planning or manipulation.
- Language Instructions Less Critical for Manipulation: For object manipulation, the agent learns action types and arguments primarily from sub-goals and action history, with language instructions playing a less significant role.
- Visual Grounding is Crucial but Challenging: The success of manipulation heavily relies on the agent's ability to correctly identify and ground action arguments to objects in the scene, highlighting visual grounding as a major challenge.
- Self-Monitoring and Backtracking Enhance Robustness: The ability to detect failures and backtrack to previous sub-goals is vital for handling unexpected situations and improving performance, especially in unseen environments.
Key Results
- HiTUT achieves state-of-the-art performance on the ALFRED benchmark, with over 160% performance gain in success rate in unseen environments compared to previous state-of-the-art.
- Even when only given the high-level goal directive (HiTUT (G Only)) without sub-goal instructions, the model achieves a 11% success rate in unseen environments, a 110% gain over MOCA.
- HiTUT outperforms previous models on almost all manipulation sub-goals by a large margin, particularly in unseen environments.
- The analysis shows that navigation actions are the hardest, with success rates significantly lower than manipulation or sub-goal planning.
- Experiments with backtracking show that increasing the number of allowed backtracks leads to higher success rates, especially in unseen environments.
Limitations and Future Work
- Object Detector Dependency: The performance of the object detector limits the model's grounding ability, as it's pre-trained and not improved during task learning. Future work could explore more robust grounding methods.
- Navigation Improvement: The paper identifies navigation as a bottleneck and suggests developing more advanced navigation modules through better structures or richer perceptions.
- Adaptive Learning: The authors emphasize the need for agents to have similar abilities to humans in diagnosing problems and requesting help when encountering difficulties, suggesting future work in interactive and active learning algorithms.
- Benchmark Design: The paper calls for a closer look at benchmark construction and objectives, questioning whether they truly reflect real-world complexity and suggesting a clearer distinction between learning tasks versus learning to follow instructions.
Summary updated