Towards Reliable Evaluation of Road Network Reconstructions

Leonardo Citraro, Mateusz Kozinski, Pascal FuaarXiv 2019 · arXiv

Introduction

Extracting road networks from aerial imagery is a critical task for autonomous driving, urban planning, and logistics. However, this is fundamentally a graph reconstruction problem rather than a simple pixel-level segmentation task. While pixel-wise accuracy is easy to measure, evaluating the topology—the underlying connectivity of the road network—remains a significant challenge. Current evaluation frameworks suffer from a "metric crisis" where popular measures often fail to penalize major topological errors, leading to inconsistent rankings of state-of-the-art algorithms.

The Metric Crisis

Existing metrics like Average Path Length Similarity (APLS) and traditional junction-based scores (JUNC) possess significant "blind spots." For instance, some metrics allow a "many-to-one" matching flaw, where multiple predicted paths can match a single ground-truth path. This allows redundant, messy reconstructions to achieve deceptively high scores. Furthermore, many metrics are insensitive to road breaks or "doubled" paths, which are catastrophic for real-world navigation.

Comparison of metric rankings across different reconstruction algorithms.

As shown in the figure above, existing metrics often disagree on which algorithm performs best. On the RoadTracer dataset, one metric might favor a segmentation-based approach while another favors a graph-based one. This inconsistency hinders research progress by making it impossible to definitively identify superior methods.

Proposed Topology-Aware Metrics

To address these limitations, this research introduces three new metrics designed to provide a more reliable and sensitive evaluation of road graphs:

  1. NEWP (Path-Based): This metric enforces a strict one-to-one correspondence between predicted and ground-truth edges. By removing an edge from the matching pool once it is "used," NEWP becomes highly sensitive to redundant or "doubled" road predictions.
  2. NEWJ (Junction-Based): Unlike traditional metrics that only consider intersections, NEWJ incorporates endpoints (dead ends) and allows matching junctions to the nearest point on an edge. This ensures that road breaks are correctly penalized.
  3. NEWG (Subgraph-Based): By sampling small subgraphs and using the Hungarian algorithm for one-to-one matching of control points, NEWG ensures that false positives located far from the ground truth are properly accounted for.

Key Results and Validation

The effectiveness of these metrics was validated using a synthetic benchmark where controlled errors (breaks, shifts, extra connections) were injected into real road maps.

Sensitivity of the proposed junction score compared to existing metrics under controlled error injection.

The experimental results demonstrate that the new metrics (NEWP, NEWJ, NEWG) are the only ones to respond consistently to all types of injected errors. While existing metrics often remained flat (indicating no perceived change in quality) as errors increased, the proposed metrics showed a clear, monotonic decrease in score. Furthermore, the new metrics exhibited significantly higher mutual correlation, providing a stable and reliable ranking of models across different datasets and urban environments.

Conclusion

Reliable evaluation is the cornerstone of algorithmic progress. By introducing metrics that are sensitive to topological integrity and consistent across different scenarios, this work provides a more robust framework for the road network extraction community. Future work may extend these "topology-aware" evaluation techniques to other curvilinear structures, such as blood vessel networks in medical imaging or neural pathways.

Summary updated

Abstract

Existing performance measures rank delineation algorithms inconsistently, which makes it difficult to decide which one is best in any given situation. We show that these inconsistencies stem from design flaws that make the metrics insensitive to whole classes of errors. To provide more reliable evaluation, we design three new metrics that are far more consistent even though they use very different approaches to comparing ground-truth and reconstructed road networks. We use both synthetic and real data to demonstrate this and advocate the use of these corrected metrics as a tool to gauge future progress.
BibTeX
@misc{citraro2019towards,
  author = {Leonardo Citraro and Mateusz Kozinski and Pascal Fua},
  title = {{Towards Reliable Evaluation of Road Network Reconstructions}},
  year = {2019},
  eprint = {1911.12467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.12467},
}
Figure