MemFusionMap: Working Memory Fusion for Online Vectorized HD Map Construction

Jingyu Song, Xudong Chen, Liupei Lu, Jie Li, Katherine A. SkinnerWACV 2025

TL;DR

This paper introduces MemFusionMap, a novel approach for constructing high-definition (HD) maps in real-time for autonomous vehicles. It tackles the challenge of complex scenarios and occlusions by introducing a "working memory" fusion module and a "temporal overlap heatmap" to better reason about historical sensor data, leading to significantly improved map accuracy.

What Problem Does This Paper Address?

Autonomous vehicles (AVs) rely heavily on high-definition (HD) maps for safe navigation and planning. These maps contain detailed information about road elements like lanes, crossings, and boundaries. While constructing these maps offline is accurate, it's expensive and inflexible. The research community is increasingly focused on online HD map construction, where AVs build these maps on-the-fly using their onboard sensors. Existing online methods, while improving, often struggle with dynamic environments, occlusions (e.g., a truck blocking a view), and effectively leveraging past sensor information. This paper aims to improve the robustness and accuracy of online vectorized HD map construction by better utilizing temporal information from a sequence of sensor inputs.

Methods

MemFusionMap builds upon existing end-to-end frameworks for vectorized HD map construction but introduces two key novel components:

  • Working Memory Fusion Module: Instead of trying to process all historical frames at once, this module maintains a "working memory" of a fixed, small number of past BEV (Bird's Eye View) features. This limited memory is designed to be more manageable and efficient. The module then fuses these working memory features with the current BEV feature. A key aspect here is the use of dilated convolutions within this fusion block, which allows the model to capture spatial context over a larger receptive field, helping it to better understand the relationships between different map elements over time.
  • Temporal Overlap Heatmap: This is a novel BEV image that explicitly informs the model about how much each grid cell in the current view has been seen in previous frames. It's initialized to all ones and then warped and updated with each new frame based on the vehicle's egomotion. A lower value in the heatmap indicates a region that is less temporally overlapped, suggesting the model should trust the current sensor data more. Conversely, a higher value suggests that historical information might be more reliable. This heatmap implicitly encodes vehicle trajectory and helps the model adapt its reasoning based on temporal redundancy.

The overall pipeline involves an image encoder, a BEV feature encoder, the proposed memory fusion block, and a transformer decoder (similar to DETR) to output the vectorized map.

Datasets

The paper evaluates MemFusionMap on two widely used autonomous driving datasets:

  • nuScenes: A multimodal dataset with synchronized sensor data from 6 cameras.
  • Argoverse2: Another comprehensive dataset featuring 7 ring cameras and 2 stereo cameras.

Crucially, the authors use "new splits" for evaluation, which are non-overlapping between training and validation sets. This is important because standard splits can be prone to overfitting, and the goal of online map construction is generalization to unseen environments.

Key Insights and Ideas

  • Limited Working Memory is Effective: The paper demonstrates that a small, fixed-size working memory (specifically 4 frames) is sufficient and even beneficial for temporal reasoning, avoiding the computational burden and potential dilution of information from trying to process too much history.
  • Explicit Temporal Overlap Cues Matter: The temporal overlap heatmap provides a simple yet powerful signal that guides the model's attention and reasoning, allowing it to adaptively leverage historical data based on how much a region has been observed.
  • Dilated Convolutions Enhance Temporal Fusion: Using dilated convolutions in the memory fusion block significantly improves the model's ability to capture spatial context and temporal relationships, leading to better map element prediction.
  • Scalability and Versatility are Key: The proposed module is designed to be easily integrated into existing pipelines by simply swapping out the temporal fusion component, making it practical for real-world deployment.

Key Results

MemFusionMap achieves state-of-the-art performance on the nuScenes and Argoverse2 benchmarks using the new, non-overlapping splits.

  • On nuScenes (60x30m range), it achieved an mAP of 38.0, outperforming StreamMapNet (34.1) by 3.9%.
  • On nuScenes (100x50m range), it achieved an mAP of 27.6, outperforming StreamMapNet (22.2) by 5.4%.
  • Similar improvements were observed on Argoverse2.
  • The method maintains a competitive runtime, comparable to StreamMapNet, indicating good efficiency.
  • Qualitative results show MemFusionMap producing more accurate map elements with better geometry and fewer missed elements, especially in challenging scenarios.

Limitations and Future Work

  • Comparison with MapTracker: While MemFusionMap outperforms StreamMapNet, it falls slightly behind MapTracker in mAP. The authors acknowledge that MapTracker benefits from additional ground truth of tracked map elements and requires time-consuming post-processing, which might limit its scalability.
  • Future Directions: The authors suggest exploring the integration of MemFusionMap with tracking paradigms like MapTracker to further enhance performance and temporal consistency. They also believe their proposed memory fusion module could be valuable for other BEV perception tasks.

Summary updated

BibTeX
@inproceedings{song2025memfusionmap,
  author = {Jingyu Song and Xudong Chen and Liupei Lu and Jie Li and Katherine A. Skinner},
  title = {{MemFusionMap: Working Memory Fusion for Online Vectorized HD Map Construction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure