Seeing the roads through the trees: A benchmark for modeling spatial dependencies with aerial imagery

Caleb Robinson, Isaac Corley, Anthony Ortiz, Rahul Dodhia, Juan M. Lavista Ferres, Peyman NajafiradarXiv 2024 · DOI · arXiv

Introduction

While humans can intuitively infer that a road continues beneath a dense patch of trees, modern artificial intelligence models frequently struggle with such "spatial reasoning." In computer vision, a prevalent "texture bias" often leads deep learning models to rely on local patterns—such as the specific color and shape of foliage—rather than the broader structural context of a scene. In the domain of aerial imagery, this results in a significant failure mode: models can identify clear, unobstructed stretches of road but fail to maintain connectivity when the road is occluded by tree canopies or shadows.

Standard semantic segmentation benchmarks often fail to explicitly test for this capability, as models can achieve high overall accuracy by simply classifying pixels based on immediate local neighbors. However, for critical geospatial applications like road network connectivity estimation or burn scar identification, models must perform long-range spatial reasoning. This paper introduces the Chesapeake Roads Spatial Context (RSC) benchmark to specifically measure how well models utilize global context to identify roads hidden under canopy.

The Chesapeake RSC Dataset

The researchers developed the Chesapeake RSC dataset to provide a rigorous testbed for spatial reasoning in overhead imagery.

Geographic distribution of the Chesapeake RSC dataset across Maryland, USA, illustrating the train, validation, and test splits.

The dataset consists of 30,000 high-resolution (1m/pixel) aerial imagery patches from the USDA’s National Agricultural Imagery Program (NAIP), covering the state of Maryland. Each 512 × 512 pixel patch is labeled with three classes: background, visible road, and "tree canopy over road" (occluded). The dataset is intentionally sampled to include challenging instances where roads are frequently interrupted by vegetation. It reflects real-world class imbalances, with occluded roads comprising only 0.7% of the total pixels, making them easy to overlook in standard metrics but vital for network integrity.

Methodology

To evaluate how current architectures handle occlusion, the authors tested several canonical semantic segmentation models:

  • Fully Convolutional Network (FCN): A baseline with a restricted receptive field (6 pixels) to establish a performance floor for local-only processing.
  • U-Net (ResNet-18/50 backbones): A standard architecture in medical and remote sensing.
  • DeepLabV3+ (ResNet-18/50 backbones): An advanced model designed for multi-scale context.

The core of the evaluation is a novel workflow that bins "tree canopy over road" pixels based on their distance to the nearest visible road segment. This allows for a precise measurement of how far a model's "spatial memory" or reasoning capability extends.

Results and Discussion

The experimental results highlight a significant performance gap between visible and occluded road detection. For instance, a U-Net with a ResNet-50 backbone achieved 84% recall on unoccluded roads but dropped to 63.5% recall for roads under canopy.

Qualitative examples of the RSC dataset. Blue segments indicate visible roads, while red segments highlight roads occluded by tree canopy, demonstrating the difficulty of maintaining connectivity.

Key findings include:

  • The "15-Pixel Wall": Across all tested models, recall on occluded roads plummeted below 50% once a pixel was more than 15 meters (15 pixels) away from a visible segment.
  • Receptive Field vs. Reasoning: Even models with large theoretical receptive fields, like DeepLabV3+, failed to effectively use that context. Having the capacity to see long-range information does not guarantee the model learns to reason with it.
  • Texture Dominance: Models remain heavily biased toward local features. If a pixel exhibits "tree-like" textures, it is typically labeled as background, even if it lies directly on a predictable road trajectory.

Conclusion

The Chesapeake RSC benchmark demonstrates that current CNN-based architectures have reached a plateau in their ability to handle long-range spatial dependencies in aerial imagery. Future work should explore architectures like Vision Transformers (ViTs) or State Space Models (SSMs), which may be better suited for capturing global context. Additionally, developing models that explicitly incorporate topological constraints—such as the inherent continuity of road networks—remains a promising frontier for geospatial AI.

Summary updated

Abstract

Fully understanding a complex high-resolution satellite or aerial imagery scene often requires spatial reasoning over a broad relevant context. The human object recognition system is able to understand object in a scene over a long-range relevant context. For example, if a human observes an aerial scene that shows sections of road broken up by tree canopy, then they will be unlikely to conclude that the road has actually been broken up into disjoint pieces by trees and instead think that the canopy of nearby trees is occluding the road. However, there is limited research being conducted to understand long-range context understanding of modern machine learning models. In this work we propose a road segmentation benchmark dataset, Chesapeake Roads Spatial Context (RSC), for evaluating the spatial long-range context understanding of geospatial machine learning models and show how commonly used semantic segmentation models can fail at this task. For example, we show that a U-Net trained to segment roads from background in aerial imagery achieves an 84% recall on unoccluded roads, but just 63.5% recall on roads covered by tree canopy despite being trained to model both the same way. We further analyze how the performance of models changes as the relevant context for a decision (unoccluded roads in our case) varies in distance. We release the code to reproduce our experiments and dataset of imagery and masks to encourage future research in this direction -- https://github.com/isaaccorley/ChesapeakeRSC.
BibTeX
@misc{robinson2024seeing,
  author = {Caleb Robinson and Isaac Corley and Anthony Ortiz and Rahul Dodhia and Juan M. Lavista Ferres and Peyman Najafirad},
  title = {{Seeing the roads through the trees: A benchmark for modeling spatial dependencies with aerial imagery}},
  year = {2024},
  eprint = {2401.06762},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.06762},
}
Figure