Abstract
Inferring street map topology from ego-view images is a fundamental challenge for autonomous driving. While traditional methods treat this as a generative task, Pix2Map reformulates it as a cross-modal retrieval problem. By learning a joint embedding space for 360° camera data and spatial graphs, the system retrieves local map layouts from a library of known road topologies. This approach ensures topological validity and outperforms state-of-the-art generative models in both seen and unseen environments.
Introduction
Autonomous vehicles (AVs) rely heavily on High-Definition (HD) maps for prior knowledge of lane boundaries, intersections, and traffic rules. However, maintaining these maps is difficult in dynamic environments or when expanding to new territories. Traditionally, "image-to-map" translation has been approached through complex generative models that map continuous image pixels to discrete, variable-sized topological graphs in a Bird’s Eye View (BEV). This is inherently difficult as it requires simultaneous learning of 3D geometry, semantic segmentation, and graph connectivity rules. Pix2Map proposes a shift in perspective: instead of generating a map from scratch, the system treats map inference as a search problem across a library of valid road layouts.
Methodology
The core of Pix2Map is a cross-modal retrieval framework that aligns visual data with spatial graph representations in a shared embedding space.
The system consists of two primary components:
- Image Encoder: Utilizing a ResNet18 backbone, this encoder processes a 360° view captured by seven "ring" cameras, condensing the visual surroundings into a 512-dimensional vector.
- Graph Encoder: A BERT-inspired Transformer treats lane nodes as tokens. Crucially, the map's adjacency matrix is used as an attention mask, forcing the model to learn features rooted in the road's specific topology.
The model is trained using a contrastive loss, maximizing similarity between paired images and ground-truth maps. To improve robustness, "Chamfer" and "Edge" losses provide "partial credit" for retrieving geometrically similar layouts even if they are not exact matches. At inference, the AV computes an image embedding and retrieves the closest graph from a pre-built library.
Datasets and Evaluation
The researchers evaluated Pix2Map using the Argoverse dataset, which provides 360° camera data and lane-level annotations for Pittsburgh and Miami. Two benchmarks were established: MapUpdate, testing on known regions at different times, and MapExpand, testing on entirely new neighborhoods.
Results and Discussion
Pix2Map significantly outperforms leading generative baselines such as TOPO-TR and PINET. In terms of Chamfer distance—a metric for geometric accuracy—Pix2Map achieved a score of 2.08, a substantial improvement over the 3.01 achieved by the best generative competitor.
Key insights from the study include:
- Topological Regularization: Unlike generative models that may produce "illegal" graphs (e.g., disconnected lanes), Pix2Map guarantees valid topology by retrieving from a library of real road layouts.
- Scalability: The system benefits from "unpaired" data; the graph library can be expanded using sources like OpenStreetMap without corresponding images, which actually improves retrieval accuracy.
- Versatility: The bidirectional nature of the embedding space allows for additional applications, such as visual localization (matching images to map segments) and Map2Pix (visualizing expected street views from map data).
Conclusion
Pix2Map demonstrates that retrieval is a powerful alternative to generation for map topology inference. While the approach is currently limited by the diversity of its graph library and can be affected by visual degradation in poor weather, it provides a more reliable foundation for AV navigation. Future work may explore hybrid models that use retrieved priors to guide generative refinement, combining the strengths of both paradigms.
Summary updated