Local Depth Edge Detection in Humans and Deep Neural Networks

Krista A. Ehinger, Erich W. Graf, Wendy J. Adams, James H. ElderWorkshops @ ICCV 2017 · OpenReview

Abstract

Distinguishing edges caused by a change in depth from other types of edges is an important problem in early vision. This study investigates the performance of humans and computer vision models on this task using spherical imagery with ground-truth LiDAR range data. By comparing various computational models for classifying depth from non-depth edges in small image patches, researchers achieved a peak performance of 86% with a convolutional neural network (CNN). Interestingly, human performance was found to be lower than the CNN, and while responses were correlated, humans and CNNs appear to weight low-level visual features differently, suggesting divergent strategies for solving the same visual problem.

Introduction

When navigating a visual environment, the human brain instantly distinguishes between the edge of a building against the sky (a depth edge) and a shadow on a sidewalk (a non-depth edge). This "edge classification" is a fundamental building block of 3D vision. However, defining ground truth for this task has historically relied on subjective human labeling. This research utilizes objective physical data to benchmark how well deep neural networks and humans can identify these boundaries using only local information.

Methodology and the SYNS Dataset

The study utilized the Southampton-York Natural Scenes (SYNS) dataset, which provides high-resolution panoramic imagery paired with precise LiDAR range measurements. By aligning 2D images with 3D depth maps, the researchers could mathematically define a "depth edge" as a physical jump in distance.

The SYNS dataset provides panoramic HDR imagery (top) paired with precise LiDAR range data (middle), enabling objective ground-truth labeling of depth edges based on physical distance.

For the computational side, a CNN was trained on 200,000 edge patches. For the human side, eight participants classified 800 of the same patches in a psychophysical experiment. To ensure a fair comparison, patches were presented at the same angular size as they would appear in a real-world scene, forcing both systems to rely strictly on local visual cues.

Comparative Performance and Insights

The CNN significantly outperformed humans on local patches, achieving 86% accuracy compared to the human average of roughly 70-75%. This suggests that local image patches contain statistical information that humans do not fully exploit in isolation.

Edge classification distinguishes between depth edges (red), where physical distance jumps, and non-depth edges (blue), such as shadows, paint, or texture changes.

Key findings regarding visual strategies include:

  • The Elevation Cue: One of the strongest predictors of a depth edge is its vertical position. Depth edges, like rooftops, are statistically more likely to occur above the horizon. The CNN exploited this "height" information effectively, whereas humans did not.
  • Feature Weighting: Humans rely heavily on luminance contrast and orientation. In contrast, the CNN was more sensitive to color cues (specifically blue-yellow contrast, likely identifying sky-object boundaries) and texture.
  • Divergent Strategies: Humans showed higher consistency with each other than with the CNN. This indicates that humans share a specific set of visual biases or heuristics that differ from the statistical shortcuts learned by the AI.

Conclusion

While CNNs currently surpass human performance in classifying edges within tiny, isolated patches, this likely reflects the CNN's ability to exploit subtle local statistics like elevation and color. In natural viewing, humans benefit from global context and eye movements that were excluded from this local task. Future work will explore how top-down scene context and more complex architectures might bridge the gap between biological and artificial vision strategies.

Summary updated

Abstract

Distinguishing edges caused by a change in depth from other types of edges is an important problem in early vision. We investigate the performance of humans and computer vision models on this task. We use spherical imagery with ground-truth LiDAR range data to build an objective ground-truth dataset for edge classification. We compare various computational models for classifying depth from non-depth edges in small images patches and achieve the best performance (86%) with a convolutional neural network. We investigate human performance on this task in a behavioral experiment and find that human performance is lower than the CNN. Although human and CNN depth responses are correlated, observers' responses are better predicted by other observers than by the CNN. The responses of CNNs and human observers also show a slightly different pattern of correlation with low-level edge cues, which suggests that CNNs and human observers may weight these features differently for classifying edges.
BibTeX
@inproceedings{ehinger2017local,
  author = {Krista A. Ehinger and Erich W. Graf and Wendy J. Adams and James H. Elder},
  title = {{Local Depth Edge Detection in Humans and Deep Neural Networks}},
  booktitle = {Workshops @ ICCV},
  year = {2017},
  url = {https://openreview.net/forum?id=3X9MNdlb7k3},
}

Loading related content...

Figure