Vehicle Behavior Prediction by Episodic-Memory Implanted NDT

Peining Shen, Jianwu Fang, Hongkai Yu, Jianru XuearXiv 2024 · arXiv

TL;DR

This paper introduces eMem-NDT (Episodic Memory implanted Neural Decision Tree), a novel framework that brings human-like explainability to vehicle behavior prediction in autonomous driving. By replacing the standard "black-box" softmax layer of deep learning models with a hierarchically clustered neural decision tree backed by an episodic memory bank, the system can explain why it predicts a certain maneuver by pointing to similar historical driving scenarios.

What Problem Does This Paper Address?

Predicting whether a surrounding vehicle will cut in, turn left, or brake suddenly (termed vehicle behavior prediction, or vBeh-Pre) is essential for safe autonomous driving. While modern deep learning models (like Transformers and LSTMs) achieve high predictive accuracy, they operate as black boxes.

If a self-driving car suddenly swerves because it predicts another vehicle will cut into its lane, human passengers and safety engineers need to know why that prediction was made. Furthermore, most existing models focus heavily on simple lane-keeping or lane-changing behaviors, ignoring fine-grained maneuvers like overtaking or decelerating straight. This paper tackles the dual challenge of achieving high-accuracy, fine-grained behavior prediction while maintaining clear, human-interpretable decision paths.

Methods

The authors propose eMem-NDT, which combines the raw representative power of deep neural networks with the structural interpretability of decision trees and human-like retrospective memory.

The eMem-NDT framework mimics human retrospective memory to explain vehicle behavior predictions.

The architecture consists of three main components:

  1. Base Feature Extractor:
  2. Stateformer: A Transformer-based model that encodes the historical trajectory and state of the target vehicle.
  3. EvolveGCN: A Graph Convolutional Network combined with an LSTM/MHA mechanism to capture the dynamic, evolving interactions between the target vehicle and surrounding road agents (pedestrians, cyclists, other cars).
  4. LLM-Guided Tree Construction: Instead of letting a decision tree grow purely based on data statistics (which is prone to overfitting), the authors use text descriptions of fine-grained vehicle behaviors (e.g., "overtaking from left"). They embed these descriptions using an OpenAI text embedding model and perform hierarchical clustering to build a natural, common-sense tree structure. Intermediate nodes are automatically labeled using GPT-4 to ensure semantic coherence from root to leaf.

  5. Episodic Memory Implantation: Each leaf node acts as an Episodic Memory Bank (EMB). During training, actual driving scenarios (trajectories and interaction graphs) corresponding to a specific behavior are distilled using a Leaf Node Memory Filtering (LNMF) algorithm to eliminate redundant, over-similar instances.

During inference, the model performs Bottom-up Memory Prototype Matching (MPM) to find how well the current scene matches the historical prototypes in the leaf nodes, followed by Top-down Leaf Link Aggregation (LLA) to output the final behavior probabilities.

Fig. 3: The construction process of eMem-NDT, which contains five steps. Commonly, the number of leaf nodes is initialized by the vehicle behavior types. Hierarchical clustering groups the text embedding of leaf nodes and makes the tree grow. In particular, we implant the episodic memories of vehicles to the correlated leaf nodes, and the leaf nodes are further optimized by measuring the feature embedding of input instances (raw vehicle states and interactions) and stored episodic memories.

Datasets

The model was evaluated on two prominent benchmarks featuring fine-grained vehicle behaviors and 3D LiDAR data:

  • BLVD: A large-scale dataset containing 77,434 vehicle instances with 13 distinct behavior labels (e.g., parallel driving, driving away to left, accelerating straight) under various lighting conditions.
  • LOKI: A highly complex dataset containing 7,491 instances across 9 behavior categories. LOKI exhibits a much smaller sample scale and a more severe class imbalance than BLVD.

Key Insights and Ideas

  • The "I've seen this before" Principle: Instead of generating an abstract probability vector, eMem-NDT justifies its prediction by retrieving a concrete, highly similar historical driving scenario from its memory bank.
  • Bridging Language and Control: Using Large Language Models (LLMs) to structure the neural decision tree ensures that the hierarchical splits correspond to human concepts (e.g., grouping "overtaking from left" and "overtaking from right" under a parent node of "Overtaking") rather than arbitrary mathematical splits.
  • Sparsity in Memory Utilization: The authors discovered that only a small proportion of memory prototypes in a leaf node actively drive a prediction. This means the memory bank can be heavily compressed (filtered) without hurting performance, keeping the model lightweight and efficient.

Key Results

The eMem-NDT model was compared against standard sequential baselines (LSTM, Transformer, CNN-BiLSTM) and standard Neural-Backed Decision Trees (NBDT).

  • Superior Accuracy: On the BLVD dataset, eMem-NDT achieved a state-of-the-art F1-score of 0.93 (Precision: 0.94, Recall: 0.92), outperforming the base black-box model (0.91) and traditional LSTMs (0.82).
  • Few-Shot Generalization: For rare, data-scarce behaviors in the BLVD dataset (like "overtaking from left" or "driving away to left"), eMem-NDT significantly outperformed NBDT, proving that the implanted episodic memory helps the model generalize from fewer examples.
  • LOKI Dataset Challenges: On the highly imbalanced LOKI dataset, eMem-NDT achieved an F1-score of 0.33. While low in absolute terms due to extreme data scarcity (some classes had fewer than 30 samples), it still matched or exceeded all deep learning baselines while providing full interpretability.

Limitations and Future Work

The primary limitation of the current work is its performance on severely imbalanced datasets like LOKI, where rare behaviors remain difficult to predict accurately across all models.

Moving forward, the authors plan to expand the capabilities of eMem-NDT beyond categorical behavior labels. They aim to explore how episodic memory can be used to interpret and predict joint vehicle trajectories and behaviors simultaneously, providing a complete, explainable lookahead path for autonomous vehicles.

Summary updated

Abstract

In autonomous driving, predicting the behavior (turning left, stopping, etc.) of target vehicles is crucial for the self-driving vehicle to make safe decisions and avoid accidents. Existing deep learning-based methods have shown excellent and accurate performance, but the black-box nature makes it untrustworthy to apply them in practical use. In this work, we explore the interpretability of behavior prediction of target vehicles by an Episodic Memory implanted Neural Decision Tree (abbrev. eMem-NDT). The structure of eMem-NDT is constructed by hierarchically clustering the text embedding of vehicle behavior descriptions. eMem-NDT is a neural-backed part of a pre-trained deep learning model by changing the soft-max layer of the deep model to eMem-NDT, for grouping and aligning the memory prototypes of the historical vehicle behavior features in training data on a neural decision tree. Each leaf node of eMem-NDT is modeled by a neural network for aligning the behavior memory prototypes. By eMem-NDT, we infer each instance in behavior prediction of vehicles by bottom-up Memory Prototype Matching (MPM) (searching the appropriate leaf node and the links to the root node) and top-down Leaf Link Aggregation (LLA) (obtaining the probability of future behaviors of vehicles for certain instances). We validate eMem-NDT on BLVD and LOKI datasets, and the results show that our model can obtain a superior performance to other methods with clear explainability. The code is available at https://github.com/JWFangit/eMem-NDT.
BibTeX
@misc{shen2024vehicle,
  author = {Peining Shen and Jianwu Fang and Hongkai Yu and Jianru Xue},
  title = {{Vehicle Behavior Prediction by Episodic-Memory Implanted NDT}},
  year = {2024},
  eprint = {2402.08423},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.08423},
}
Figure