Towards Understanding End-of-trip Instructions in a Taxi Ride Scenario

Deepthi Karkada, Ramesh Manuvirakurike, Kallirroi Georgila

TL;DR

This paper introduces a novel dataset and annotation scheme for understanding natural language instructions given to a taxi driver at the end of a ride. They tackle the challenge of grounding these instructions in visual scenes, enabling machines to interpret phrases like "stop next to the blue car" by identifying referents and their spatial relationships.

What Problem Does This Paper Address?

Imagine you're a passenger in a taxi, and you need to tell the driver exactly where to stop. This isn't just about saying "stop here"; it involves complex human-like understanding. You might say, "Drop me off in front of the second black pickup truck" or "Park next to the big tree." For a human driver, this is intuitive. They can visually identify the "second black pickup truck" or the "big tree" (the referent) and then understand the spatial relationship ("in front of," "next to") to pinpoint the exact stopping location.

The challenge for AI is to replicate this. It requires bridging the gap between natural language descriptions and visual scenes. Specifically, this paper addresses the problem of:

  1. Visual Reference Resolution: Identifying the specific object(s) (referents) a user is talking about within a visual scene, especially when there are many similar objects (distractors).
  2. Directional Description Understanding: Interpreting spatial prepositions and phrases (e.g., "in front of," "behind," "next to") to understand the target location relative to the referent.
  3. Action Identification: Recognizing the user's intent (e.g., "stop," "drop me off").

This is crucial for developing more natural and intuitive human-robot interaction, particularly in autonomous driving or navigation systems.

Methods

The core of this work lies in creating a dataset and an annotation scheme to facilitate the development of AI systems capable of understanding these end-of-trip instructions.

  • Data Collection: They used a crowdsourcing paradigm where Amazon Mechanical Turk workers were shown images and asked to provide natural language descriptions of how to stop at a designated "red cross" target location. Turkers were instructed to imagine they were passengers in a taxi.
  • Annotation Scheme: The natural language descriptions were annotated with three key components:
  • Action (ACT): The verb indicating the desired operation (e.g., "stop," "park").
  • Referent (REF): The object(s) or vehicle(s) used as a landmark (e.g., "the blue car," "the big tree").
  • Directional Description (DD): Phrases indicating the target location's position relative to the referent (e.g., "in front of," "next to," "behind").
  • Visual Annotations: Beyond language, the images were also annotated with:
  • Ground truth labels and coordinates for all objects.
  • Dimensions of vehicles and objects.
  • Coordinates of the target location.
  • Distance and direction of the target location from objects, quantified using polar coordinates (r, θ), where 'r' is the distance and 'θ' is the angle. This (r, θ) representation is a novel way to encode spatial relationships.
  • Referent Identification Model: For the task of identifying the referent from a description, they explored several methods:
  • Random Baseline: Randomly picking an object.
  • Sub-string Matching: Comparing the user's description with ground truth object labels.
  • Sentence Embeddings: Using pre-trained sentence embeddings (Wiki-unigram) to represent both the referent description and object labels, then finding the best match using dot products.

Datasets

The paper introduces a novel dataset comprising:

  • Synthetic Images: Programmatically generated top-down views of street scenes with vehicles and objects. These are used to control for complexities like visual segmentation and object identification, focusing purely on language-vision grounding.
  • Real-World Images: Street-view images extracted from Google Maps. These provide more diverse and complex visual environments.

The dataset contains a total of 324 synthetic images and 68 real-world images, with a significant number of user descriptions and tokens.

Key Insights and Ideas

  • (r, θ) for Spatial Encoding: Quantifying the target location relative to a referent using polar coordinates (r, θ) is a novel and effective way to represent spatial relationships, allowing for direct prediction of target coordinates.
  • Importance of Context: The study highlights how crucial visual context is for resolving ambiguous language. Even simple phrases require understanding the surrounding objects.
  • Diversity in Referent Descriptions: Real-world images lead to more diverse and complex referent descriptions (e.g., mentioning car brands, stickers), posing a greater challenge than synthetic scenes.
  • Sentence Embeddings Outperform Baselines: Using pre-trained sentence embeddings significantly improves the accuracy of referent identification compared to simpler methods like sub-string matching. This suggests that capturing semantic similarity is key.
  • Inter-Annotator Agreement: The high inter-annotator agreement (Cohen's kappa of 0.81) for their annotation scheme indicates its robustness and clarity.

Key Results

  • Referent Identification Accuracy: The sentence embedding model achieved a 70.2% accuracy in identifying the correct referent on synthetic images, significantly outperforming the random baseline (6.25%) and sub-string matching (47.5%).
  • Annotation Statistics: The dataset contains a substantial number of descriptions and annotations, with a notable difference in the complexity of referent descriptions between synthetic and real-world images.
  • Inter-Rater Reliability: A high Cohen's kappa score of 0.81 was achieved, demonstrating the reliability of their annotation process.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Handling Ambiguity with Multiple Similar Objects: The current model struggles when multiple similar objects are present in the scene, making it hard to pinpoint the exact referent. They suggest that conversational dialogue between the driver and rider could help resolve such ambiguities.
  • Plural Referents: The model has difficulty with plural descriptions (e.g., "between the two blue cars") and currently resolves ties randomly.
  • Real-World Complexity: Real-world descriptions are more elaborate, and the model needs to be extended to handle this complexity more robustly.
  • Full Pipeline Automation: The current work focuses on referent identification. Future work aims to automate the entire pipeline, including action and directional description parsing, to achieve end-to-end target location identification.
  • Dialogue Integration: Incorporating dialogue data is a key future step to address ambiguity and improve interaction.
  • ISO-Space Framework: Investigating the applicability of existing spatial annotation frameworks like ISO-Space to their domain is also a promising avenue.

Summary updated

BibTeX
@inproceedings{karkada2018towards,
  author = {Deepthi Karkada and Ramesh Manuvirakurike and Kallirroi Georgila},
  title = {{Towards Understanding End-of-trip Instructions in a Taxi Ride Scenario}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure