Towards Reliable Evaluation of Algorithms for Road Network Reconstruction from Aerial Images

Leonardo Citraro, Mateusz Kozinski, Pascal FuaECCV 2020 · DOI

Abstract

Reconstructing road networks as graphs from aerial imagery is a fundamental challenge in remote sensing. However, progress is hindered by evaluation metrics that are often inconsistent and "blind" to critical topological errors. This paper identifies design flaws in existing metrics and introduces three new, more sensitive alternatives: OPT-P, OPT-J, and OPT-G. These metrics provide a more reliable and consistent framework for assessing the quality of reconstructed road networks, aligning more closely with the manual effort required for map correction.

Introduction

Extracting road networks as graph representations (nodes and edges) from satellite imagery is a classic computer vision problem. While spatial accuracy—determining if a road is in the correct location—is straightforward to measure, evaluating topology—the connectivity and structure of the network—is significantly more complex. Comparing two graphs is an NP-complete problem, leading researchers to rely on proxy metrics.

A major issue in the field is that these proxies are often inconsistent. An algorithm may rank highly according to one metric but fail under another. This inconsistency stems from "blind spots" in existing metrics that make them insensitive to specific errors, such as small gaps or spurious roads, which would be time-consuming for a human operator to fix.

The Topology Evaluation Crisis

The authors highlight that popular metrics like APLS (Average Path Length Similarity) and TLTS (Topological Lane-level Topology Similarity) are often insensitive to "overconnections" or spurious roads. If an algorithm predicts extra roads that do not change the shortest path between two points, these metrics may not penalize the error. For real-world applications, such errors are costly, as they require manual intervention to prune the map.

Proposed Metrics: OPT-P, OPT-J, and OPT-G

To address these flaws, the authors propose three new metrics designed to be sensitive to all types of topological discrepancies.

OPT-P (Path-Based)

Unlike previous path metrics that only compare shortest path lengths, OPT-P uses an iterative sampling scheme. It ensures that no two sampled paths share edges and that trajectories must remain close to the ground truth. This makes it sensitive to spurious roads that previous metrics ignored.

Comparison of path-based metrics: APLS and TLTS (left) fail to penalize spurious roads, whereas OPT-P (right) correctly identifies and penalizes the discrepancy.

OPT-J (Junction-Based)

Traditional junction metrics often only consider intersections with three or more roads. OPT-J improves upon this by including endpoints as features and allowing matching a feature in one graph to an edge in another. This prevents the metric from "forgiving" road breaks that occur in the middle of a segment.

The OPT-J junction score incorporates endpoints and edge matching to ensure that topological breaks are adequately penalized.

OPT-G (Subgraph-Based)

Previous subgraph metrics typically sampled only from the ground truth, missing false positives in the prediction. OPT-G samples from both graphs and uses the Hungarian algorithm to enforce a strict one-to-one matching of control points, ensuring that errors like "doubled" or parallel roads are properly penalized.

Key Results and Insights

The authors evaluated their metrics using synthetic benchmarks with controlled errors, as well as the RoadTracer and DeepGlobe datasets.

  • Ranking Stability: When comparing state-of-the-art algorithms, old metrics gave conflicting rankings. The new OPT metrics consistently ranked RCNNUNet as the top performer across different datasets.
  • Correlation and Sensitivity: The three new metrics correlate much more strongly with each other than the old metrics do, suggesting they are successfully measuring the same underlying "quality" of the reconstruction.
  • Human-Centric Evaluation: A core argument of the paper is that a good metric should reflect the effort required for a human to fix the map. The OPT metrics are specifically designed to capture this "cost of repair."

Conclusion

The proposed OPT metrics provide a more robust and reliable way to evaluate road network reconstruction algorithms. By addressing the "blind spots" of previous methods, they offer a clearer benchmark for progress in the field. The authors have made their code publicly available to encourage adoption and standardization.

Summary updated

BibTeX
@inproceedings{citraro2020towards,
  author = {Leonardo Citraro and Mateusz Kozinski and Pascal Fua},
  title = {{Towards Reliable Evaluation of Algorithms for Road Network Reconstruction from Aerial Images}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58604-1\_42},
}
Figure