TL;DR
This paper introduces SPINE, an online semantic planner that uses Large Language Models (LLMs) to interpret incomplete natural language mission instructions for robots operating in unknown environments. SPINE enables robots to autonomously infer subtasks, map their surroundings, and validate plans in real-time, significantly improving efficiency and reducing user intervention compared to existing methods.
What Problem Does This Paper Address?
Imagine telling a robot, "Communications are down. Why?" This simple request implies a complex mission: the robot needs to explore, identify communication infrastructure, assess damage, and report findings. Current robotic planning systems struggle with such "incomplete" specifications because they often require pre-built maps and highly detailed, explicit instructions. This is a significant limitation, as real-world environments are rarely fully mapped, and users prefer to interact with robots using natural, high-level language. Existing LLM-based planners, while powerful, often assume complete maps or well-defined missions, making them unsuitable for dynamic, partially known environments. SPINE aims to bridge this gap by enabling robots to handle ambiguous instructions and explore unknown territories autonomously.
Methods
SPINE is built around a novel architecture that combines an LLM with a receding horizon planning framework. Here's how it works:
- LLM-Powered Plan Generation: At its core, SPINE uses an LLM to interpret the natural language mission. The LLM is prompted with the mission, the current state of the robot's knowledge (represented as a semantic graph), and a history of actions. It then reasons about implied subtasks, such as navigation, active mapping, and user interaction, and generates a sequence of atomic behaviors.
- Semantic Mapping: A key component is an open-vocabulary semantic mapper that builds a topological graph of the environment. This graph represents regions (traversable areas) and objects, enriched with semantic information. This mapper can be initialized with prior data (like satellite imagery) but is designed to be updated online as the robot explores.
- Online Validation: LLMs can sometimes "hallucinate" or make errors. SPINE addresses this with a crucial plan validation module. This module checks if the LLM's proposed actions are syntactically correct, physically reachable within the current map, and if exploration goals are feasible. If an action is invalid, feedback is sent back to the LLM to refine the plan.
- Receding Horizon Execution: The generated plan is executed one step at a time. As the robot gathers new information (through its sensors and the semantic mapper), the plan is continuously updated and refined, allowing for adaptation to unforeseen circumstances.
Datasets
The paper does not mention specific datasets used for training LLMs. However, the evaluation is performed in:
- Simulated Environments: A large-scale rural outdoor space (over 40,000m²) and a cluttered outdoor environment (over 20,000m²).
- Real-World Environments: An office park.
Prior maps for these environments were constructed using satellite imagery or UAV-generated maps, highlighting their inherent incompleteness and inaccuracies.
Key Insights and Ideas
- LLMs as Implicit Task Decomposers: The most significant insight is that LLMs can effectively infer complex, multi-step subtasks from vague natural language instructions, acting as a powerful "implicit task decomposer" for robots.
- Online Validation is Crucial for LLM Planners: LLMs are prone to errors, especially in dynamic environments. The validation module is not just a safety feature; it's essential for making LLM-driven planning robust and reliable in real-world scenarios. It prevents the robot from executing unsafe or impossible actions.
- Incomplete Maps are the Norm, Not the Exception: The paper strongly emphasizes that robots must operate effectively with incomplete prior knowledge. SPINE's ability to map and plan online is key to its practical applicability.
- Efficiency Gains from Integrated Planning and Mapping: By integrating online mapping with LLM-driven planning and validation, SPINE achieves significant improvements in time and distance compared to methods that first map completely and then plan.
Key Results
SPINE demonstrated strong performance in both simulation and real-world experiments:
- Efficiency: SPINE was over twice as efficient in terms of time and distance compared to an LLM-as-planner baseline that required a full prior map.
- User Interaction: SPINE required significantly less user interaction than explicit tasking baselines.
- Mission Success: In simulation, SPINE achieved a 94.3% mission success rate, comparable to baselines that received more complete information. In real-world tests, its success rate was even higher than in simulation.
- Validation Impact: An ablation study showed that validation is increasingly important as the uncertainty in the prior map increases, significantly improving mission success rates.
Limitations and Future Work
- Internet Dependency: SPINE currently relies on an internet connection for LLM queries, which is a limitation for operation in environments without connectivity.
- Single-Robot Focus: The current work focuses on single-robot planning.
Future work suggested by the authors includes:
- Onboard LLMs: Adapting smaller, open-source LLMs to run directly on the robot for offline operation.
- Multi-Robot Planning: Extending SPINE to support online and distributed multi-robot planning applications.