Abstract
In recent years, we have seen significant steps taken in the development of self-driving cars. Multiple companies are starting to roll out impressive systems that work in a variety of settings. These systems can sometimes give the impression that full self-driving is just around the corner and that we would soon build cars without even a steering wheel. The increase in the level of autonomy and control given to an AI provides an opportunity for new modes of human-vehicle interaction. However, surveys have shown that giving more control to an AI in self-driving cars is accompanied by a degree of uneasiness by passengers. In an attempt to alleviate this issue, recent works have taken a natural language-oriented approach by allowing the passenger to give commands that refer to specific objects in the visual scene. Nevertheless, this is only half the task as the car should also understand the physical destination of the command, which is what we focus on in this paper. We propose an extension in which we annotate the 3D destination that the car needs to reach after executing the given command and evaluate multiple different baselines on predicting this destination location. Additionally, we introduce a model that outperforms the prior works adapted for this particular setting.
1. Introduction
While the dream of fully autonomous vehicles often focuses on the "driving" part—staying in lanes and avoiding obstacles—the "interaction" part is just as critical. If you tell a robot taxi, "Drop me off near that silver SUV," the car needs to do more than just find the SUV; it needs to determine a safe, logical physical coordinate to stop.
In their AAAI-22 paper, Grujicic et al. tackle this gap between natural language understanding and physical world navigation. They address the "last-mile" problem: predicting the exact 3D destination within the immediate observable vicinity based on a spontaneous passenger command.
2. The Talk2Car-Destination Dataset
To address this challenge, the researchers introduced Talk2Car-Destination, an expansion of the original Talk2Car dataset built on nuScenes. This dataset bridges the gap between high-level navigation and low-level object detection by mapping natural language commands to 3D physical destinations.
- Scale: 8,301 training, 1,159 validation, and 2,439 test samples.
- Annotation: Each command was annotated by three different people to indicate the physical end position of the car in a top-down view.
- Diversity: The dataset includes various weather conditions (rain, sun) and lighting (day, night) across Boston and Singapore.
3. Methodology
The researchers developed a decoupled pipeline to move from a raw image and text command to a 3D destination:
- 3D Object Detection: The authors opted for a monocular vision-only detector (FCOS3D) rather than LiDAR-based methods. They found that monocular vision handled long-distance objects more gracefully than LiDAR, which struggled with temporal/spatial parallax and range drop-offs beyond 70 meters.
- Referred Object Detector: This module uses Sentence-BERT to embed the command and matches it against the features of detected 3D objects to find the "target" (e.g., the specific SUV mentioned).
- Layout Encoding: The model creates a top-down "bird's-eye view" (BEV) representation of the scene, including road geometry, the ego-car's position, and the 3D locations of all detected objects.
- The PDPC Model: Instead of predicting a single point, the Physical Destination Predictor for Commands (PDPC) uses a Feature Pyramid Network (FPN) to process the BEV layout. It predicts a Gaussian Mixture Model (GMM) distribution across the spatial grid, allowing it to represent complex, multi-modal possibilities and express uncertainty.
4. Results and Discussion
The PDPC model set a new state-of-the-art for this task, achieving a Prediction Accuracy (PA4) of 49.51%—a 32% relative improvement over the best-adapted baseline. The Median Displacement Error (MDE) was reduced to 4.39 meters.
Key insights from the study include:
- The "Referred Object" is the North Star: Knowing which object the passenger is talking about is the single biggest factor in accuracy. Without this, performance drops by nearly 50%.
- Monocular > LiDAR for Grounding: Monocular 3D detection proved more robust for linguistic grounding, avoiding sensor lag issues and handling long-distance objects better.
- Uncertainty Matters: By predicting a distribution rather than a single point, the model can express ambiguity in commands, providing safer inputs for downstream navigation.
5. Conclusion
While promising, the model currently operates on static snapshots and does not yet integrate dynamic environment changes or traffic rule constraints (e.g., avoiding stopping in intersections). Future work points toward end-to-end multi-task models that allow destination intent to inform object detection directly, moving closer to a seamless human-vehicle interaction experience.
Summary updated