TL;DR
This paper introduces eMem-NDT (Episodic Memory implanted Neural Decision Tree), a novel framework that brings human-like explainability to vehicle behavior prediction in autonomous driving. By replacing the standard "black-box" softmax layer of deep learning models with a hierarchically clustered neural decision tree backed by an episodic memory bank, the system can explain why it predicts a certain maneuver by pointing to similar historical driving scenarios.
What Problem Does This Paper Address?
Predicting whether a surrounding vehicle will cut in, turn left, or brake suddenly (termed vehicle behavior prediction, or vBeh-Pre) is essential for safe autonomous driving. While modern deep learning models (like Transformers and LSTMs) achieve high predictive accuracy, they operate as black boxes.
If a self-driving car suddenly swerves because it predicts another vehicle will cut into its lane, human passengers and safety engineers need to know why that prediction was made. Furthermore, most existing models focus heavily on simple lane-keeping or lane-changing behaviors, ignoring fine-grained maneuvers like overtaking or decelerating straight. This paper tackles the dual challenge of achieving high-accuracy, fine-grained behavior prediction while maintaining clear, human-interpretable decision paths.
Methods
The authors propose eMem-NDT, which combines the raw representative power of deep neural networks with the structural interpretability of decision trees and human-like retrospective memory.
The architecture consists of three main components:
- Base Feature Extractor:
- Stateformer: A Transformer-based model that encodes the historical trajectory and state of the target vehicle.
- EvolveGCN: A Graph Convolutional Network combined with an LSTM/MHA mechanism to capture the dynamic, evolving interactions between the target vehicle and surrounding road agents (pedestrians, cyclists, other cars).
-
LLM-Guided Tree Construction: Instead of letting a decision tree grow purely based on data statistics (which is prone to overfitting), the authors use text descriptions of fine-grained vehicle behaviors (e.g., "overtaking from left"). They embed these descriptions using an OpenAI text embedding model and perform hierarchical clustering to build a natural, common-sense tree structure. Intermediate nodes are automatically labeled using GPT-4 to ensure semantic coherence from root to leaf.
-
Episodic Memory Implantation: Each leaf node acts as an Episodic Memory Bank (EMB). During training, actual driving scenarios (trajectories and interaction graphs) corresponding to a specific behavior are distilled using a Leaf Node Memory Filtering (LNMF) algorithm to eliminate redundant, over-similar instances.
During inference, the model performs Bottom-up Memory Prototype Matching (MPM) to find how well the current scene matches the historical prototypes in the leaf nodes, followed by Top-down Leaf Link Aggregation (LLA) to output the final behavior probabilities.
Datasets
The model was evaluated on two prominent benchmarks featuring fine-grained vehicle behaviors and 3D LiDAR data:
- BLVD: A large-scale dataset containing 77,434 vehicle instances with 13 distinct behavior labels (e.g., parallel driving, driving away to left, accelerating straight) under various lighting conditions.
- LOKI: A highly complex dataset containing 7,491 instances across 9 behavior categories. LOKI exhibits a much smaller sample scale and a more severe class imbalance than BLVD.
Key Insights and Ideas
- The "I've seen this before" Principle: Instead of generating an abstract probability vector, eMem-NDT justifies its prediction by retrieving a concrete, highly similar historical driving scenario from its memory bank.
- Bridging Language and Control: Using Large Language Models (LLMs) to structure the neural decision tree ensures that the hierarchical splits correspond to human concepts (e.g., grouping "overtaking from left" and "overtaking from right" under a parent node of "Overtaking") rather than arbitrary mathematical splits.
- Sparsity in Memory Utilization: The authors discovered that only a small proportion of memory prototypes in a leaf node actively drive a prediction. This means the memory bank can be heavily compressed (filtered) without hurting performance, keeping the model lightweight and efficient.
Key Results
The eMem-NDT model was compared against standard sequential baselines (LSTM, Transformer, CNN-BiLSTM) and standard Neural-Backed Decision Trees (NBDT).
- Superior Accuracy: On the BLVD dataset, eMem-NDT achieved a state-of-the-art F1-score of 0.93 (Precision: 0.94, Recall: 0.92), outperforming the base black-box model (0.91) and traditional LSTMs (0.82).
- Few-Shot Generalization: For rare, data-scarce behaviors in the BLVD dataset (like "overtaking from left" or "driving away to left"), eMem-NDT significantly outperformed NBDT, proving that the implanted episodic memory helps the model generalize from fewer examples.
- LOKI Dataset Challenges: On the highly imbalanced LOKI dataset, eMem-NDT achieved an F1-score of 0.33. While low in absolute terms due to extreme data scarcity (some classes had fewer than 30 samples), it still matched or exceeded all deep learning baselines while providing full interpretability.
Limitations and Future Work
The primary limitation of the current work is its performance on severely imbalanced datasets like LOKI, where rare behaviors remain difficult to predict accurately across all models.
Moving forward, the authors plan to expand the capabilities of eMem-NDT beyond categorical behavior labels. They aim to explore how episodic memory can be used to interpret and predict joint vehicle trajectories and behaviors simultaneously, providing a complete, explainable lookahead path for autonomous vehicles.
Summary updated