Introduction
Most current Embodied AI tasks focus on short-term goals, such as navigating to a single visible object. However, real-world robotics requires long-horizon reasoning. A robot tasked with tidying a home must explore unknown rooms, remember object locations, manage carrying capacity, and plan efficient paths over thousands of timesteps. Existing end-to-end Reinforcement Learning (RL) approaches often fail in these scenarios due to a lack of exploration guarantees, difficulty in backtracking, and the vanishing gradient problem in long-horizon planning.
The Hierarchical Transport Policy (HTP)
To address these challenges, researchers introduced the Hierarchical Transport Policy (HTP), a modular framework that decomposes the complex transport task into specialized components managed by a high-level finite-state machine.
The HTP architecture consists of several key modules:
- Topological Graph Builder: This module constructs a map of the environment using 'nodes' that store visual features and 'closeness scores' for objects. This allows the agent to maintain a persistent memory of the scene.
- High-Level Controller: Acting as the central 'brain,' this controller decides which behavior to trigger—
Explore,Pickup[Object], orDrop—based on the current state and graph information. - Pick-Drop Module: This module handles the logic for approaching and interacting with objects once they are identified in the topological graph.
Strategic Exploration and Navigation
Efficient exploration is critical for finding objects in large, unknown environments. HTP utilizes a Weighted Frontier Exploration strategy. The agent identifies 'frontiers' (boundaries between explored and unexplored space) and uses a Graph Convolutional Network (GCN) to predict an Exploration Score. This score prioritizes frontiers most likely to lead to target objects, mimicking human-like search patterns.
For navigation, HTP employs a hybrid approach. It uses classical motion planning ($A^*$) to traverse known space efficiently and a specialized RL-based 'Object Nav' policy for the 'last mile'—the precise approach and grasping of an object. This combination leverages the reliability of geometric planning with the semantic flexibility of RL.
Experimental Evaluation
The framework was evaluated using the Habitat simulator with Matterport3D (MP3D) scans. The researchers introduced the Long-HOT benchmark, where an agent must transport $K$ objects to a goal under load constraints. The benchmark also includes 'container' objects that increase the agent's carrying capacity, adding a layer of strategic decision-making. Additionally, the model was tested on the MultiOn benchmark to demonstrate its versatility.
Results and Generalization
HTP demonstrated significant performance gains over traditional end-to-end models:
- Success Rate: On the standard Long-HOT task, HTP achieved a 59% success rate, compared to 43% for the 'NoMap' RL baseline.
- Scalability: In large-scale environments (dimensions > 40m), the RL baseline's performance dropped to near zero (2.8%), while HTP maintained a 22% success rate, showcasing superior generalization.
- Efficiency: HTP set a new state-of-the-art on the MultiOn benchmark, improving both success rates and path efficiency (SPL).
Conclusion
Long-HOT and the HTP framework represent a significant step toward robust long-horizon robotics. By combining modular high-level control with topological memory and strategic exploration, HTP overcomes the limitations of end-to-end RL in complex, large-scale environments. Future work will focus on improving robustness to sensor noise and handling more complex obstacles and view-dependent object detection.
Summary updated