Abstract
Natural language route instructions are often "underspecified," omitting common-sense actions that humans perform intuitively. This paper presents MARCO, an agent designed to follow human directions by inferring implicit actions through a combination of linguistic parsing and spatial reasoning. MARCO achieves a 61% success rate in complex virtual environments, approaching the 69% human baseline. Ablation studies demonstrate that the ability to infer implicit turns and travel actions is critical, as removing these capabilities causes performance to drop significantly from 61% to 28%.
Introduction
When providing directions, humans rarely offer exhaustive step-by-step manuals. Instead, we provide "skeletal" plans, assuming the listener can fill in gaps—such as turning to face a hallway before walking down it or traveling to a landmark before making a turn. For traditional robotic agents, this lack of explicit detail often leads to failure. This work addresses the challenge of building an agent that can recover these missing pieces by integrating linguistic knowledge with environmental context.
The MARCO Architecture
The MARCO agent utilizes a dual-stack architecture to bridge the gap between words and actions.
The Linguistic Stack
The linguistic stack transforms raw text into executable specifications. It consists of three primary components:
- Syntax Parser: Employs a probabilistic context-free grammar to identify verb-argument structures.
- Content Framer: Maps syntax into a semantic "content frame" and leverages WordNet to resolve synonyms (e.g., equating "futon" with "couch").
- Instruction Modeler: The core reasoning engine that converts frames into Compound Action Specifications. It applies spatial logic to infer necessary preconditions, such as the requirement to face a path before traversing it.
The Execution Stack
The execution stack handles the real-time interaction with the environment. The Executor interleaves action and perception using primitives like Turn, Travel, and Verify. A View Description Matcher compares symbolic descriptions (e.g., "a green hallway") against sensory data. Notably, MARCO can perform exploratory actions; if an instruction is ambiguous, the agent may look in multiple directions to determine the most likely path.
Experimental Evaluation
The system was evaluated using a corpus of 786 natural language instructions generated by human directors across three large-scale virtual indoor environments. To isolate navigation logic from parsing errors, MARCO was tested using hand-verified "gold-standard" parses. Performance was compared against a baseline of 36 human followers.
Results and Discussion
MARCO demonstrated a high degree of competence, reaching the intended destination in 61% of trials, compared to 69% for human followers. The agent's performance correlated strongly with the quality of the instructions; it succeeded in 84% of cases with "good" instructions but struggled with poor ones.
Ablation studies highlighted the necessity of action inference. When the ability to infer implicit actions was disabled, MARCO's success rate plummeted to 28%. Specifically, removing implicit turns reduced success to 32%, while removing implicit travel actions reduced it to 42%. These results confirm that instructions are not complete programs but skeletons that require contextual "common sense" to execute.
Conclusion and Future Work
MARCO successfully demonstrates that agents can follow underspecified natural language instructions by inferring implicit actions. While the current system relies on symbolic perception and verified parses, future work aims to integrate raw computer vision and automated speech-to-text. The researchers suggest that this framework of skeletal plan expansion could be applied to other domains, such as following assembly instructions or recipes.
Summary updated