Introduction
As autonomous vehicle (AV) technology matures, the focus shifts from basic navigation to nuanced human-robot interaction. A critical challenge in this domain is the "last-mile" problem: understanding natural language "end-of-trip" instructions. Unlike standard navigation tasks that rely on precise GPS coordinates, passengers typically provide situated, relative commands such as "stop just past the blue car" or "drop me off in front of the second black pickup truck." This paper introduces a novel dataset and a mathematical framework to bridge the gap between human linguistic cues and robotic spatial navigation.
Data Collection and Annotation
To capture the diversity of human instructions, the authors crowdsourced descriptions using Amazon Mechanical Turk. Participants were presented with images containing a target drop-off location (marked with a red cross) and asked to provide instructions for a hypothetical taxi driver.
The resulting dataset comprises 1,281 descriptions across 392 images, split between synthetic top-down street views and real-world Google Street View captures. The synthetic environment allows for a clean evaluation of linguistic resolution without the noise of complex computer vision tasks like occlusion. Instructions are annotated using a three-part linguistic scheme: Actions (e.g., "park"), Referents (REF, e.g., "the cop car"), and Directional Descriptions (DD, e.g., "in front of").
Mathematical Modeling of Spatial Language
A key contribution of this work is the mapping of fuzzy directional phrases to a physical coordinate system. The authors propose an $r\theta$ (polar coordinate) model where the target location is defined by a distance ($r$) and an angle ($\theta$) relative to the visual referent.
This model quantifies spatial language; for instance, "next to" typically maps to a smaller $r$ and $\theta$ compared to "just past." This mathematical bridge allows the system to translate a phrase like "behind the truck" into a specific physical zone for the vehicle to target.
Referent Identification and Results
The authors established baseline models for the task of referent identification—determining which object in a scene the user is referring to. They compared three primary approaches:
- Random Baseline: Achieved 6.25% accuracy.
- Sub-string Matching: Matched words in the instruction to object labels, achieving 47.5% accuracy.
- Sentence Embeddings: Used
sent2vecto compute similarity between descriptions and labels.
The results demonstrated that pre-trained Wiki-unigram embeddings significantly outperformed models trained only on the in-domain dataset, reaching 70.2% accuracy. This suggests that the broad vocabulary used by humans (e.g., swapping "cop car" for "police car") requires the semantic depth of large-scale general corpora to resolve effectively.
Conclusion and Future Work
While the $r\theta$ model and embedding-based referent resolution provide a strong foundation, challenges remain. Current models struggle with ambiguity, such as scenes containing multiple identical objects or plural referents. Future research will focus on integrating dialogue capabilities to allow the AV to ask clarifying questions and automating the full pipeline from raw pixels and text to final navigation coordinates.
Summary updated