Agent Prioritization for Autonomous Navigation

Khaled S. Refaat, Kai Ding, Natalia Ponomareva, Stéphane Ross

Abstract

In autonomous navigation, a planning system reasons about other agents to plan a safe and plausible trajectory. Before planning starts, agents are typically processed with computationally intensive models for recognition, tracking, motion estimation and prediction. With limited computational resources and a large number of agents to process in real time, it becomes important to efficiently rank agents according to their impact on the decision making process. This allows spending more time processing the most important agents. We propose a system to rank agents around an autonomous vehicle (AV) in real time. We automatically generate a ranking data set by running the planner in simulation on real-world logged data, where we can afford to run more accurate and expensive models on all the agents. The causes of various planner actions are logged and used for assigning ground truth importance scores. The generated data set can be used to learn ranking models. In particular, we show the utility of combining learned features, via a convolutional neural network, with engineered features designed to capture domain knowledge. We show the benefits of various design choices experimentally. When tested on real AVs, our system demonstrates the capability of understanding complex driving situations.

The Challenge of Attention in Autonomous Driving

Autonomous vehicles (AVs) often face "information overload" in crowded urban environments. In these settings, processing every nearby pedestrian and car with high-fidelity models is computationally prohibitive. Driving is essentially a game of attention; humans instinctively ignore a car parked several blocks away while focusing intensely on a cyclist about to merge.

For an AV, every detected object requires computation for tracking, intent prediction, and motion estimation. Running expensive deep-learning models, such as LSTMs for trajectory prediction, on dozens of agents in real-time creates a significant bottleneck. To address this, we propose a "triage" system: a lightweight model that ranks agents by their potential impact on the AV’s path, ensuring limited computational resources are spent on the most critical threats.

The Prioritization Pipeline

The autonomous navigation pipeline highlighting the prioritization stage.

As shown in the system architecture, prioritization acts as a filter between initial perception and the more intensive agent processing and planning stages. By ranking agents immediately after perception, the system can selectively apply high-fidelity models only to the most relevant actors in the scene.

Methodology: A Hybrid Ranking Approach

Our system utilizes a hybrid ranking model that combines the strengths of deep learning with classical machine learning and domain expertise:

  1. Data Generation (The "Oracle"): To train the model, we ran a "heavy" planner in simulation on real-world driving logs. This offline planner uses high-accuracy models for every agent. By observing which agents actually caused the AV to brake or nudge, we automatically generated ground-truth "importance scores" without human labeling.
  2. The CNN (Spatial Context): A Convolutional Neural Network (CNN) processes top-down images of the scene. Instead of a standard video feed, we use a "Time Image" representation that encodes temporal dynamics into separate image channels.
  3. The GBDT (Domain Knowledge): We combine the CNN’s features with a Gradient Boosted Decision Tree (GBDT) fed with hand-engineered features, such as "distance to the AV’s planned trajectory" or "time to collision."
  4. Pairwise Ranking: Rather than predicting an absolute score for each agent, the model is trained to compare pairs of agents to determine which is more important. This technique, common in search engine ranking, proves highly effective for navigation.

Representing Time Spatially

The "Time Image" representation encoding temporal dynamics and lane information into image channels.

Instead of using slow Recurrent Neural Networks (RNNs), we represent time as pixel intensity or separate channels in a top-down map. This allows the model to "see" motion, velocity, and lane context in a single forward pass of a CNN, making it significantly more efficient for real-time applications.

Key Results and Insights

Our experiments revealed several critical findings:

  • Hybrid Superiority: The CNN excels at capturing complex spatial interactions, while engineered features in the GBDT are better at capturing "hard" physics like time-to-collision. Using both outperformed either model alone.
  • Accuracy Gains: The learned models outperformed standard heuristics (like ranking by proximity) by 21% to 45% based on the NDCG metric. The hybrid model placed the "most relevant" agent in the top spot 86.88% of the time.
  • Real-Time Performance: The GBDT component can rank thousands of agents in just 0.5 milliseconds, fitting perfectly within the tight loops of autonomous driving software.

Conclusion and Future Work

While highly effective, the current system has limitations, such as a spatial range constraint of 80x80 meters for the CNN component. Future work will explore "planner-agnostic" data generation to ensure the ranking model remains robust even as the underlying planning software evolves. By efficiently managing computational attention, this prioritization framework allows AVs to navigate complex, agent-dense environments with greater safety and reliability.

Summary updated

Abstract

In autonomous navigation, a planning system reasons about other agents to plan a safe and plausible trajectory. Before planning starts, agents are typically processed with computationally intensive models for recognition, tracking, motion estimation and prediction. With limited computational resources and a large number of agents to process in real time, it becomes important to efficiently rank agents according to their impact on the decision making process. This allows spending more time processing the most important agents. We propose a system to rank agents around an autonomous vehicle (AV) in real time. We automatically generate a ranking data set by running the planner in simulation on real-world logged data, where we can afford to run more accurate and expensive models on all the agents. The causes of various planner actions are logged and used for assigning ground truth importance scores. The generated data set can be used to learn ranking models. In particular, we show the utility of combining learned features, via a convolutional neural network, with engineered features designed to capture domain knowledge. We show the benefits of various design choices experimentally. When tested on real AVs, our system demonstrates the capability of understanding complex driving situations.
BibTeX
@inproceedings{refaat2019agent,
  author = {Khaled S. Refaat and Kai Ding and Natalia Ponomareva and Stéphane Ross},
  title = {{Agent Prioritization for Autonomous Navigation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=lk9aNISRVrq},
}
Figure