TL;DR
This paper introduces a novel dataset of Japanese route descriptions generated from real-world maps, aiming to improve language-based navigation for autonomous systems. They found that while route and destination information are crucial for clarity, starting point information is frequently omitted, posing a challenge for natural language understanding in navigation.
What Problem Does This Paper Address?
Conveying route information accurately, especially for autonomous systems, is a complex task. It requires not only specifying the start, end, and path but also leveraging contextual cues like landmarks. Languages like Japanese, which often omit explicit subjects, further complicate generating clear navigational instructions. Existing research has explored direction-giving conversations and spatial reasoning, but there's a gap in collecting and analyzing natural language route descriptions derived from real-world maps, specifically focusing on how different components of route information (start, path, destination) affect clarity. This paper addresses this gap by creating a dataset of Japanese route descriptions and analyzing their clarity.
Methods
The researchers collected route information reference expressions by having participants describe routes on 20 different real-world Japanese maps. They used Yahoo! Crowd Sourcing for data collection. Participants were instructed to use relative directional terms ("front, back, left, right") from their current perspective on the map, avoiding absolute directions (north, south, east, west) or cardinal directions. The collected expressions were then manually annotated to determine the presence of information about the starting point (X), route (Y), and destination (Z). Additionally, a clarity rating was obtained for each expression through a separate survey.
Datasets
The study introduced a new dataset of 1600 Japanese route information reference expressions. This dataset was collected using 20 real-world maps, with each map having two starting and ending point patterns, resulting in 40 stimuli per map. The expressions were generated by 40 individuals per route. This dataset differs from previous work by using real-world maps rather than fictional environments and by focusing on collecting route descriptions rather than general spatial interactions.
Key Insights and Ideas
- Starting Point Omission is Common: A significant portion of the collected expressions (37.5%) omitted information about the starting point, highlighting a challenge in how people naturally describe routes.
- Route and Destination are Key for Clarity: Expressions that included information about the route and destination were generally clearer than those that did not.
- Misidentification Hurts Clarity: When participants mistakenly identified the starting and ending points (misidentification W), the clarity of their expressions significantly decreased.
- Real-World Maps Matter: Using real-world maps, with their inherent complexities and multiple route options, provides a more realistic scenario for collecting navigation instructions compared to abstract or fictional environments.
Key Results
- Clarity Ratings: The average clarity rating for expressions that correctly identified the start and end points (W=FALSE) was 2.79, significantly higher than those with misidentification (W=TRUE) at 2.07.
- Component Impact on Clarity:
- Starting point (X) explanations, though often omitted, slightly improved clarity when present (2.82 vs. 2.72 for absent).
- Route (Y) explanations had a substantial positive impact on clarity (2.79 vs. 2.00 for absent).
- Destination (Z) explanations also significantly improved clarity (2.79 vs. 2.43 for absent).
- Predominance of Route and Destination: Route (Y) and destination (Z) explanations were present in the vast majority of expressions (93.9%).
Limitations and Future Work
The authors acknowledge that while relative reference expressions are inherently complex and can involve information from three or more points, obtaining such utterances was not always possible. Future work will involve annotating the collected data based on the double-cross model, which is designed to handle spatial relations involving three or more points. The researchers also plan to make the collected data, including map information, classifications, and clarity ratings, publicly accessible.
Summary updated