MemRoadNet: Human-like Memory Integration for Free Road Space Detection

Sidra Shafiq, Abdullah Aman Khan, Jie ShaoSensors 2025 · DOI

TL;DR

This paper introduces MemRoadNet, a novel deep learning framework for free road space detection in autonomous driving. It uniquely integrates a human-inspired "Memory Bank" system that learns from past driving experiences, allowing it to achieve state-of-the-art performance using only RGB images, rivaling more complex multimodal systems.

What Problem Does This Paper Address?

Autonomous driving systems need to accurately identify drivable road areas, a task crucial for navigation and safety. Current methods, primarily Convolutional Neural Networks (CNNs), process each input image independently. This means they don't learn from past driving experiences, making them less adaptable to diverse conditions like varying lighting, weather, and road layouts. While multimodal approaches (using LiDAR, depth sensors, etc.) improve performance, they are computationally expensive and complex. This paper addresses the gap of leveraging "Accumulated Experiential Knowledge" (AEK) in single-modality (RGB-only) systems to achieve robust and efficient road detection.

Methods

MemRoadNet combines a powerful InternImage-XL backbone for feature extraction with a UPerNet decoder for multi-scale feature fusion. The core innovation is the Human-like Memory Bank system, inspired by cognitive neuroscience. This system comprises three subsystems:

  • Episodic Memory: Stores specific past driving "experiences" (compressed features, contextual info, performance outcomes).
  • Semantic Memory: Organizes generalized knowledge patterns.
  • Working Memory: Maintains recent contextual information for immediate use.

Crucially, experiences are assigned an "emotional valence" based on segmentation performance (e.g., high IoU means positive valence). This allows the system to prioritize learning from successful and unsuccessful predictions. During training and inference, the Memory Bank retrieves relevant past experiences using a sophisticated similarity metric (combining pattern, context, recency, and importance) and integrates them into the current feature representations via an attention mechanism. This "memory-guided feature enhancement" allows the model to compensate for limitations of single-modality input.

Datasets

The framework was trained and evaluated on a combination of datasets:

  • Virtual KITTI 2: Synthetic dataset with controlled variations for training the memory system.
  • KITTI road: Real-world dataset for evaluating performance on urban driving scenarios.
  • Cityscapes: Large-scale dataset with diverse urban environments.
  • R2D: Synthetic dataset with surface normal information, used to test cross-domain generalization.

Key Insights and Ideas

  • AEK is crucial for robustness: By learning from past experiences, the model becomes significantly more adaptable to challenging and varied driving conditions, even with just RGB input.
  • "Emotional valence" as a learning signal: Assigning performance-based valences to memories allows the system to intelligently prioritize learning from critical successes and failures, mimicking human learning.
  • Single-modality can rival multimodal: The memory system effectively bridges the information gap typically filled by multiple sensors, achieving competitive results without the added complexity and cost.
  • Human-inspired memory architecture works: The integration of episodic, semantic, and working memory, along with consolidation and forgetting mechanisms, proves effective for long-term knowledge accumulation and retrieval.

Key Results

  • State-of-the-art single-modality performance: MemRoadNet achieved a MaxF score of 0.9666 on the KITTI road benchmark, outperforming existing single-modality methods.
  • Competitive with multimodal systems: The framework achieved performance comparable to leading multimodal approaches on the KITTI road dataset, despite using only RGB input.
  • Robust generalization: The model demonstrated strong performance on synthetic (R2D) and diverse urban (Cityscapes) datasets, indicating good cross-domain adaptation.
  • Efficiency: While requiring more parameters and FLOPs than some lightweight models, its computational efficiency is competitive with other high-performing methods, and it achieves a respectable inference speed of 18.5 FPS.

Limitations and Future Work

  • Inference Speed: The current 18.5 FPS might be insufficient for ultra-high-speed applications, and evaluation on embedded hardware is needed.
  • Catastrophic Forgetting: The fixed memory capacity with importance-based forgetting could lead to forgetting rare but critical edge cases during long-term deployment.
  • Generalization to Unseen Conditions: Performance might degrade in extreme weather or novel infrastructure not present in training data.

Future work includes exploring domain adaptation techniques, active learning, memory compression, adaptive capacity management, meta-learning for hyperparameter optimization, and extending memory mechanisms to multi-task learning.

Summary updated

Abstract

Detecting available road space is a fundamental task for autonomous driving vehicles, requiring robust image feature extraction methods that operate reliably across diverse sensor-captured scenarios. However, existing approaches process each input independently without leveraging Accumulated Experiential Knowledge (AEK), limiting their adaptability and reliability. In order to explore the impact of AEK, we introduce MemRoadNet, a Memory-Augmented (MA) semantic segmentation framework that integrates human-inspired cognitive architectures with deep-learning models for free road space detection. Our approach combines an InternImage-XL backbone with a UPerNet decoder and a Human-like Memory Bank system implementing episodic, semantic, and working memory subsystems. The memory system stores road experiences with emotional valences based on segmentation performance, enabling intelligent retrieval and integration of relevant historical patterns during training and inference. Experimental validation on the KITTI road, Cityscapes, and R2D benchmarks demonstrates that our single-modality RGB approach achieves competitive performance with complex multimodal systems while maintaining computational efficiency and achieving top performance among single-modality methods. The MA framework represents a significant advancement in sensor-based computer vision systems, bridging computational efficiency and segmentation quality for autonomous driving applications.
BibTeX
@inproceedings{shafiq2025memroadnet,
  author = {Sidra Shafiq and Abdullah Aman Khan and Jie Shao},
  title = {{MemRoadNet: Human-like Memory Integration for Free Road Space Detection}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25216600},
}
Figure