Predicting Physical World Destinations for Commands Given to Self-Driving Cars

Dusan Grujicic, Thierry Deruyttere, Marie-Francine Moens, Matthew B. Blaschko

Abstract

In recent years, we have seen significant steps taken in the development of self-driving cars. Multiple companies are starting to roll out impressive systems that work in a variety of settings. These systems can sometimes give the impression that full self-driving is just around the corner and that we would soon build cars without even a steering wheel. The increase in the level of autonomy and control given to an AI provides an opportunity for new modes of human-vehicle interaction. However, surveys have shown that giving more control to an AI in self-driving cars is accompanied by a degree of uneasiness by passengers. In an attempt to alleviate this issue, recent works have taken a natural language-oriented approach by allowing the passenger to give commands that refer to specific objects in the visual scene. Nevertheless, this is only half the task as the car should also understand the physical destination of the command, which is what we focus on in this paper. We propose an extension in which we annotate the 3D destination that the car needs to reach after executing the given command and evaluate multiple different baselines on predicting this destination location. Additionally, we introduce a model that outperforms the prior works adapted for this particular setting.

Introduction: Bridging Language and 3D Navigation

While self-driving cars are becoming increasingly autonomous, passengers often feel a sense of unease when they lack direct control. Natural language interfaces can alleviate this, but a significant technical gap remains: most current systems focus on either high-level city navigation or low-level object detection.

The challenge lies in spatial language understanding. When a passenger says, "Let me out near my friend in the red shirt," the car must perform a complex chain of reasoning: identify the "friend," understand the 3D geometry of the scene, and determine a safe, logical physical destination. This paper tackles the "low-level" problem of predicting the exact 3D end-goal within the car's immediate observable vicinity.

Frontal (a) and top-down (b) views of a command execution scene, showing the referred object and the predicted destination heatmap.

The PDPC Architecture

We propose the Physical Destination Predictor for Commands (PDPC), a modular architecture that decouples object identification from destination prediction:

  1. 3D Object Detection: The system uses FCOS3D, a monocular 3D detector, to identify objects in the frontal camera view. Interestingly, vision-only models performed better than LIDAR-based ones at long distances (>70m) for this specific task.
  2. Referred Object Detection: A sub-module combines Sentence-BERT command embeddings with visual features to identify which specific object the passenger is talking about.
  3. Layout Encoding: The scene is converted into a multi-channel top-down "layout tensor" representing the road, the car's current position, and the locations of all detected objects.
A top-down layout encoding representing the road geometry, sidewalks, and detected object bounding boxes.
  1. Destination Predictor: This core module uses a Feature Pyramid Network (FPN) to extract features at multiple scales. Instead of predicting a single point, it predicts a mixture of Gaussians across a spatial grid. This allows the model to handle uncertainty and represent multiple potential destinations. During inference, "Top-K sharpening" filters for the top 32 components to remove outliers and ensure the car stays on-road.

Experimental Setup: Talk2Car-Destination

We introduce Talk2Car-Destination, an extension of the Talk2Car dataset built on the nuScenes autonomous driving framework. It includes 8,301 training, 1,159 validation, and 2,439 test samples across urban environments in Boston and Singapore. Each command is paired with three human-annotated 3D destinations and an "intent" label (e.g., Park, Stop, Follow, Overtake).

Results and Discussion

PDPC was evaluated using Average Displacement Error (ADE), Median Displacement Error (MDE), and Prediction Accuracy (PA) at 2m and 4m thresholds.

  • Accuracy: PDPC achieved a Median Displacement Error (MDE) of 4.39 meters, outperforming the best baseline (Endpoint VAE) at 5.22 meters.
  • Precision: In terms of PA₂ (predicting within 2 meters of the goal), PDPC showed a 32% relative improvement over the strongest baselines.
  • Intent Performance: The model was particularly strong on "Pick Up" and "Stop" commands, achieving median errors as low as 2.89 meters.

Key insights include the critical importance of visual grounding; when provided with ground-truth object locations, error dropped significantly. Furthermore, modeling the destination as a probability distribution rather than a single coordinate allowed the car to represent uncertainty more effectively.

Conclusion and Future Directions

PDPC successfully translates natural language into precise 3D navigation goals. Future work will explore incorporating raw visual features from the frontal camera directly into the predictor and training the pipeline in an end-to-end fashion. Additionally, integrating this static destination prediction with dynamic path planning to avoid moving obstacles is a necessary step for real-world deployment.

Summary updated

Abstract

In recent years, we have seen significant steps taken in the development of self-driving cars. Multiple companies are starting to roll out impressive systems that work in a variety of settings. These systems can sometimes give the impression that full self-driving is just around the corner and that we would soon build cars without even a steering wheel. The increase in the level of autonomy and control given to an AI provides an opportunity for new modes of human-vehicle interaction. However, surveys have shown that giving more control to an AI in self-driving cars is accompanied by a degree of uneasiness by passengers. In an attempt to alleviate this issue, recent works have taken a natural language-oriented approach by allowing the passenger to give commands that refer to specific objects in the visual scene. Nevertheless, this is only half the task as the car should also understand the physical destination of the command, which is what we focus on in this paper. We propose an extension in which we annotate the 3D destination that the car needs to reach after executing the given command and evaluate multiple different baselines on predicting this destination location. Additionally, we introduce a model that outperforms the prior works adapted for this particular setting.
BibTeX
@inproceedings{grujicic2022predicting,
  author = {Dusan Grujicic and Thierry Deruyttere and Marie-Francine Moens and Matthew B. Blaschko},
  title = {{Predicting Physical World Destinations for Commands Given to Self-Driving Cars}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=qKU6pCvHPo},
}
Figure