RESPOND: Risk-Enhanced Structured Pattern for LLM-driven Online Node-level Decision-making

Dan Chen, Heye Huang, Tiantian Chen, Zheng Li, Yongji Li, Yuhui Xu, Sikai ChenarXiv 2025 · arXiv · OpenReview

TL;DR

This paper introduces RESPOND, a novel framework for LLM-driven autonomous driving that tackles the limitations of unstructured memory by using a structured, risk-aware representation. RESPOND achieves superior safety and learning efficiency through a hybrid decision-making pipeline and a unique "one-crash-to-generalize" reflection mechanism, outperforming existing methods in simulation and demonstrating real-world applicability.

What Problem Does This Paper Address?

Current LLM-based autonomous driving systems often struggle with decision-making in complex, safety-critical scenarios. They typically rely on unstructured, plain-text memories, which leads to two main issues:

  1. Imprecise Scene Retrieval: Semantic matching of text descriptions can fail to distinguish subtle but crucial differences in high-risk situations, leading to incorrect or delayed actions.
  2. Inefficient Learning from Failures: When these systems make mistakes (e.g., cause a collision), extracting reusable knowledge from unstructured text logs is difficult, making it hard to learn from failures and generalize to similar situations.

Furthermore, existing approaches often lack a robust way to encode the nuanced risk perception that human drivers use, especially concerning spatial relationships and potential hazards. This gap hinders the development of truly safe, adaptive, and personalized autonomous driving systems.

Methods

RESPOND addresses these challenges with a multi-faceted approach:

  • Structured Risk-Environment Encoding: Instead of plain text, RESPOND represents each driving scene as a unified 5x3 ego-centric matrix. This matrix encodes spatial topology, road constraints, and crucially, quantified risk levels (derived from the Driver Risk Field - DRF). This structured format allows for precise retrieval of "risk patterns."
  • Two-Tier Memory System:
    • Layer 1 (Exact Pattern Memory): Stores exact matches of the 15D vectorized risk patterns and their corresponding safe actions. This enables rapid, high-confidence action reuse in familiar, high-risk situations.
  • Layer 2 (Sub-Pattern Abstraction Memory): Decomposes the risk grid into semantic sub-patterns (e.g., "front risk," "left risk"). This allows for more flexible decision-making and personalized style adaptation in lower-risk contexts, and importantly, enables generalization.
  • Hybrid Rule+LLM Decision Pipeline: This pipeline balances safety and adaptability. In high-risk scenarios, it prioritizes exact memory reuse or uses sub-patterns to constrain LLM reasoning. In low-risk scenarios, it allows for personalized style adaptation guided by sub-patterns.
  • Pattern-Aware Reflection Mechanism: This is a key innovation for learning efficiency. Upon detecting a failure (collision or safety violation), the system analyzes the pre- and post-failure frames. It extracts tactical corrections and updates both Layer 1 (exact action) and Layer 2 (abstract sub-patterns) memories. This "one-crash-to-generalize" capability allows the system to learn from a single failure and avoid repeating similar mistakes, significantly accelerating learning.
  • Human-in-the-Loop Personalization: For low-risk scenarios, users can provide feedback to shape the driving style (e.g., "Sporty" or "Comfortable"). This feedback is abstracted into Layer 2 sub-patterns, enabling data-efficient personalization without retraining.

Datasets

  • highway-env simulator: Used for controlled closed-loop simulations to evaluate safety, learning efficiency, and robustness across various traffic densities and lane configurations.
  • HighD dataset: A large-scale, real-world dataset of naturalistic highway trajectories. Used for evaluating the system's robustness and decision-making capability in unseen, safety-critical scenarios without prior memory matches.

Key Insights and Ideas

  • Structured Risk Representation is Key: Moving beyond plain text to a structured, quantitative risk representation (the 5x3 matrix) is crucial for precise memory retrieval and robust decision-making in safety-critical driving.
  • "One-Crash-to-Generalize" is Achievable: The pattern-aware reflection mechanism, by updating both exact and abstract memory layers, demonstrates that significant learning can occur from a single failure, drastically improving learning efficiency.
  • Layered Memory for Dual Goals: The two-tier memory (exact recall for safety, abstract patterns for generalization and personalization) effectively balances the need for deterministic safety in critical situations with flexibility and adaptation.
  • Human-like Risk Perception: The DRF-based risk encoding allows the system to mimic human risk perception, leading to more intuitive and safer decisions.
  • Personalization without Retraining: The human-in-the-loop personalization, by leveraging Layer 2 sub-patterns, allows for efficient and safe style adaptation with minimal human input.

Key Results

  • Simulation Performance: RESPOND (L1+L2) achieved a 100% Success Rate (SR) in standard highway simulations, significantly outperforming state-of-the-art LLM-based (DiLu) and RL-based (GRAD) agents.
  • Robustness: Even in the most challenging, high-density scenarios (Lane-5, Density 3.0), RESPOND (L1+L2) maintained a robust 70% SR, with a significantly smaller performance drop compared to baselines.
  • Learning Efficiency: RESPOND achieved a ~7x improvement in learning efficiency, reaching 100% SR after only 6 collisions, compared to DiLu requiring ~40 collisions.
  • Generalization: The system eliminated 100% of recurring longitudinal (FRONT/REAR) collisions and 97% of lateral (LEFT/RIGHT) collisions after a single reflective update.
  • Real-World Validation (HighD): In 53 high-risk cut-in scenarios, RESPOND's decisions were judged safer than recorded human actions in 84.9% of cases, demonstrating practical relevance and capability in unseen, real-world situations.
  • Personalization: The "Sporty" mode, trained with only 20 human feedback steps, achieved an 80% SR, demonstrating efficient and safe style adaptation.

Limitations and Future Work

  • Simulator-to-Real-World Gap: While RESPOND showed strong zero-shot performance on HighD, some discrepancies were noted due to simplifications in the DRF formulation (e.g., not fully accounting for truck length) and differences in real-world traffic characteristics not modeled in simulation. The authors acknowledge this is a known limitation of transferring simulator-calibrated configurations.
  • Future Work: The paper suggests that future work could involve adapting the system for real-world data without additional tuning, further exploring the integration of human intuition with structured knowledge representation, and developing more sophisticated methods for handling long-tail scenarios and edge cases. The authors also highlight the potential for RESPOND as a tool for offline safety analysis and edge-case mining.

Summary updated

Abstract

Current LLM-based driving agents that rely on unstructured plain-text memory suffer from low-precision scene retrieval and inefficient reflection. To address this limitation, we present RESPOND, a structured decision-making framework for LLM-driven agents grounded in explicit risk patterns. RESPOND represents each ego-centric scene using a unified 5 by 3 matrix that encodes spatial topology and road constraints, enabling consistent and reliable retrieval of spatial risk configurations. Based on this representation, a hybrid rule and LLM decision pipeline is developed with a two-tier memory mechanism. In high-risk contexts, exact pattern matching enables rapid and safe reuse of verified actions, while in low-risk contexts, sub-pattern matching supports personalized driving style adaptation. In addition, a pattern-aware reflection mechanism abstracts tactical corrections from crash and near-miss frames to update structured memory, achieving one-crash-to-generalize learning. Extensive experiments demonstrate the effectiveness of RESPOND. In highway-env, RESPOND outperforms state-of-the-art LLM-based and reinforcement learning based driving agents while producing substantially fewer collisions. With step-wise human feedback, the agent acquires a Sporty driving style within approximately 20 decision steps through sub-pattern abstraction. For real-world validation, RESPOND is evaluated on 53 high-risk cut-in scenarios extracted from the HighD dataset. For each event, intervention is applied immediately before the cut-in and RESPOND re-decides the driving action. Compared to recorded human behavior, RESPOND reduces subsequent risk in 84.9 percent of scenarios, demonstrating its practical feasibility under real-world driving conditions. These results highlight RESPONDs potential for autonomous driving, personalized driving assistance, and proactive hazard mitigation.
BibTeX
@misc{chen2025respond,
  author = {Dan Chen and Heye Huang and Tiantian Chen and Zheng Li and Yongji Li and Yuhui Xu and Sikai Chen},
  title = {{RESPOND: Risk-Enhanced Structured Pattern for LLM-driven Online Node-level Decision-making}},
  year = {2025},
  eprint = {2512.20179},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.20179},
}
Figure