Introduction
Extracting road networks from aerial imagery is a critical task for autonomous driving, urban planning, and logistics. However, this is fundamentally a graph reconstruction problem rather than a simple pixel-level segmentation task. While pixel-wise accuracy is easy to measure, evaluating the topology—the underlying connectivity of the road network—remains a significant challenge. Current evaluation frameworks suffer from a "metric crisis" where popular measures often fail to penalize major topological errors, leading to inconsistent rankings of state-of-the-art algorithms.
The Metric Crisis
Existing metrics like Average Path Length Similarity (APLS) and traditional junction-based scores (JUNC) possess significant "blind spots." For instance, some metrics allow a "many-to-one" matching flaw, where multiple predicted paths can match a single ground-truth path. This allows redundant, messy reconstructions to achieve deceptively high scores. Furthermore, many metrics are insensitive to road breaks or "doubled" paths, which are catastrophic for real-world navigation.
As shown in the figure above, existing metrics often disagree on which algorithm performs best. On the RoadTracer dataset, one metric might favor a segmentation-based approach while another favors a graph-based one. This inconsistency hinders research progress by making it impossible to definitively identify superior methods.
Proposed Topology-Aware Metrics
To address these limitations, this research introduces three new metrics designed to provide a more reliable and sensitive evaluation of road graphs:
- NEWP (Path-Based): This metric enforces a strict one-to-one correspondence between predicted and ground-truth edges. By removing an edge from the matching pool once it is "used," NEWP becomes highly sensitive to redundant or "doubled" road predictions.
- NEWJ (Junction-Based): Unlike traditional metrics that only consider intersections, NEWJ incorporates endpoints (dead ends) and allows matching junctions to the nearest point on an edge. This ensures that road breaks are correctly penalized.
- NEWG (Subgraph-Based): By sampling small subgraphs and using the Hungarian algorithm for one-to-one matching of control points, NEWG ensures that false positives located far from the ground truth are properly accounted for.
Key Results and Validation
The effectiveness of these metrics was validated using a synthetic benchmark where controlled errors (breaks, shifts, extra connections) were injected into real road maps.
The experimental results demonstrate that the new metrics (NEWP, NEWJ, NEWG) are the only ones to respond consistently to all types of injected errors. While existing metrics often remained flat (indicating no perceived change in quality) as errors increased, the proposed metrics showed a clear, monotonic decrease in score. Furthermore, the new metrics exhibited significantly higher mutual correlation, providing a stable and reliable ranking of models across different datasets and urban environments.
Conclusion
Reliable evaluation is the cornerstone of algorithmic progress. By introducing metrics that are sensitive to topological integrity and consistent across different scenarios, this work provides a more robust framework for the road network extraction community. Future work may extend these "topology-aware" evaluation techniques to other curvilinear structures, such as blood vessel networks in medical imaging or neural pathways.
Summary updated