Abstract
A long-term goal of artificial intelligence is to have an agent execute commands communicated through natural language. In many cases the commands are grounded in a visual environment shared by the human who gives the command and the agent. Execution of the command then requires mapping the command into the physical visual space, after which the appropriate action can be taken. In this paper we consider the former. Or more specifically, we consider the problem in an autonomous driving setting, where a passenger requests an action that can be associated with an object found in a street scene. Our work presents the Talk2Car dataset, which is the first object referral dataset that contains commands written in natural language for self-driving cars. We provide a detailed comparison with related datasets such as ReferIt, RefCOCO, RefCOCO+, RefCOCOg, Cityscape-Ref and CLEVR-Ref. Additionally, we include a performance analysis using strong state-of-the-art models. The results show that the proposed object referral task is a challenging one for which the models show promising results but still require additional research in natural language processing, computer vision and the intersection of these fields. The dataset can be found on our website: http://macchina-ai.eu/
Introduction
While autonomous vehicles (AVs) are advancing rapidly, they often lack a "human-in-the-loop" interface that feels natural to passengers. In complex urban environments, passengers may feel unsettled by a lack of control or find the vehicle's behavior hesitant. To address this, systems must be able to interpret free-form natural language commands and map them to specific objects in a 3D scene.
Existing object referral datasets like RefCOCO or ReferIt focus on static image descriptions rather than action-oriented commands. They also lack a "car-centric" perspective, failing to account for the specific spatial perspectives and object categories encountered on the road. The Talk2Car dataset fills this gap. Built on top of the nuScenes dataset, it contains 11,959 commands for 850 videos, including rich metadata like LIDAR, RADAR, and GPS. Unlike descriptive datasets, Talk2Car uses unconstrained "command" language that often includes social context and complex coreference resolution.
Methods
The researchers evaluated two main categories of state-of-the-art models on the Talk2Car benchmark:
- Region Proposal Based Methods: Models such as OSM and SCRC utilize a Single Shot Detector (SSD) to identify potential objects (regions of interest). Neural networks then score these regions based on their match with the natural language command.
- Non-Region Proposal Based Methods: Models like MAC and Stack-NMN avoid pre-extracted boxes. They use attention mechanisms to decompose commands into reasoning steps, analyzing the image globally to locate the target.
Additionally, three simple baselines were implemented to establish task difficulty: Random Selection (RS), Biggest Overlapping Bounding Box (BOBB) (exploiting spatial bias), and Random Noun Matching (RNM).
Key Insights and Results
The evaluation revealed several critical findings about the intersection of NLP and computer vision in driving contexts:
- Reasoning is Essential: The MAC (Memory, Attention, and Composition) model was the top performer, achieving a 50.51% [email protected]. Its success suggests that the ability to perform multiple reasoning steps to disambiguate objects (e.g., distinguishing "the car" from "the silver car next to the lamppost") is vital.
- Efficiency: MAC demonstrated an inference time of 51ms on a Titan XP, indicating its potential for near-real-time application.
- The Challenge of Distance: Models struggle significantly with objects located far from the vehicle. This is a critical hurdle, as humans often give commands early to allow the vehicle time to prepare for a maneuver.
- Command Complexity: Performance generally decreased as command length increased, as longer sentences often introduced "noise" or irrelevant social information that confused the models.
Conclusion
Talk2Car establishes a challenging, non-trivial benchmark for the community. While current models show promise, significant work remains. Future research could integrate the dataset's untapped LIDAR and RADAR modalities to improve distance estimation and object disambiguation. Furthermore, moving from single-object grounding to multi-object scenarios and interactive dialogue—where the car can ask clarifying questions—represents the next frontier in creating truly intuitive human-AV interfaces.
Summary updated