Abstract
Thierry Deruyttere, Simon Vandenhende, Dusan Grujicic, Luc Van Gool, Marie-Francine Moens. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
As autonomous vehicles (AVs) advance, a significant barrier to their widespread adoption remains the "black box" nature of their decision-making processes. To foster trust and usability, passengers must be able to communicate naturally with the vehicle—for instance, by requesting to "Park in the shade over there" or "Pick up my friend standing by the bus stop." This necessitates solving the object referral task: given an image $I$ and a natural language command $C$, the system must identify the specific region $R$ referenced by the human.
Existing datasets like RefCOCO or CLEVR-Ref often rely on synthetic shapes or lack the linguistic nuances of real-world driving, which involve complex spatial relationships and social context. Talk2Car addresses this by introducing the first large-scale object referral dataset specifically designed for autonomous driving.
The Talk2Car Dataset
The Talk2Car dataset is built upon the nuScenes dataset, leveraging high-quality sensor data from urban environments in Boston and Singapore. It consists of 11,959 natural language commands across 850 videos. Unlike previous datasets, Talk2Car features:
- Action-Oriented Commands: Commands are free-form and focus on vehicle control rather than static descriptions.
- Linguistic Complexity: Commands average 11 words and often include "noise" or irrelevant social context (e.g., "My friend is getting out of the car... Stop and let me out too!").
- Diverse Scenarios: The dataset includes 23 object classes and varies in depth and scene density.
Methods
The authors established baselines using seven distinct models categorized into three groups:
- Simple Baselines: Including random selection, center-bias (BOBB), and Random Noun Matching (RNM) which aligns nouns in commands with object categories.
- Region Proposal Based: These models utilize a pre-trained SSD-512 detector to identify candidate objects.
- Object Sentence Mapping (OSM): Maps regions and text into a shared embedding space.
- Spatial Context Recurrent ConvNet (SCRC): Enhances local features with 8-dimensional spatial coordinates to improve positioning accuracy.
- Modular/Non-Region Proposal Based:
- Memory, Attention, and Composition (MAC): A VQA-style model that performs multi-step reasoning over image and text.
- Stack-NMN: A modular network that decomposes commands into logical sub-tasks.
Key Insights and Results
The evaluation revealed several critical findings regarding the state of vision-language models in driving contexts:
- Spatial Reasoning is Essential: The SCRC model outperformed the OSM model by 3.4%, demonstrating that explicit spatial features are crucial for disambiguating objects (e.g., "the car on the left").
- The Noun Hurdle: The RNM baseline achieved ~10% accuracy, suggesting that simply identifying the correct object category is a significant initial challenge.
- Ambiguity and Reasoning: Performance drops as command length increases or as the number of similar objects in a scene grows. The MAC model emerged as the top performer, achieving 50.51% [email protected]. Its multi-step reasoning allowed it to remain more stable in ambiguous scenes compared to simpler models. Furthermore, MAC proved efficient for real-time applications with an inference speed of 51ms.
Conclusion and Future Work
Talk2Car highlights a significant gap between current vision-language models and the requirements for safe vehicle control. While models like MAC show promise, several areas for future research remain:
- 3D Grounding: Moving beyond 2D images to incorporate LIDAR and RADAR data for better spatial grounding.
- Interactive Dialogue: Transitioning from one-way commands to two-way dialogue where the car can seek clarification.
- Action Execution: Bridging the gap between identifying an object and planning a safe driving trajectory based on that grounding.
Summary updated