Automatic Sky View Factor Estimation from Street View Photographs—A Big Data Approach

Jianming Liang, Jianhua Gong, Jun Sun, Jieping Zhou, Wenhang Li, Yi Li, Liu Jin, Shen ShenRemote 2017 · DOI

TL;DR

This paper introduces a novel, big-data approach to automatically estimate Sky View Factor (SVF) using readily available street view panoramas. By leveraging deep learning for sky segmentation and a robust stitching pipeline, the method offers a cost-effective and scalable alternative to traditional SVF estimation techniques that rely on expensive LiDAR or aerial photogrammetry data.

What Problem Does This Paper Address?

Sky View Factor (SVF) is a crucial parameter for understanding urban climate and informing urban planning, as it quantifies the fraction of the sky visible from a given point. Traditionally, SVF estimation relies on hemispherical (fisheye) photography or high-resolution 3D urban models derived from LiDAR or aerial photogrammetry. However, these methods are often limited by the availability and high cost of acquiring such detailed spatial data, especially for large urban areas. The proliferation of street view imagery (e.g., Google Street View, Baidu Street View) presents a massive, globally accessible dataset. The challenge is to effectively and accurately extract SVF from these panoramas, which are not inherently designed for this purpose and require significant processing to reconstruct the necessary 3D view.

Methods

The paper proposes a four-step workflow for automatic SVF estimation from street view images:

  1. Street View Image Retrieval: Images are fetched from street view providers (GSV, BSV, TSV) using their respective APIs. This involves specifying location, field of view (FOV), heading, and pitch.
  2. Panorama Stitching: Multiple retrieved images are stitched together to reconstruct a full 360-degree panorama. This is a non-trivial step as it requires transforming individual images back into an equirectangular projection, accounting for view and projection transformations. The authors implemented this in C++.
  3. Panorama Classification using Deep Learning: A pre-trained SegNet deep learning model is employed for pixel-wise semantic segmentation of the reconstructed panoramas. SegNet classifies road scenes into 12 classes, which are then simplified into "sky" and "non-sky" categories for SVF calculation. The model was adapted to handle 1024x1024 panoramic images.
  4. Hemispherical Transformation and SVF Calculation: The stitched and classified panoramas are transformed into fisheye images. SVF is then calculated by summing the contributions of sky pixels, weighted by their angular distribution, using a formula that approximates the transformation from an equisoild angle projection to an equal-areal projection. The Lambert's cosine law is intentionally omitted to simplify the analysis.

Datasets

  • Google Street View (GSV): Used for data retrieval and application in Manhattan.
  • Baidu Street View (BSV) and Tencent Street View (TSV): Used for data retrieval and comparison with OAP3D.
  • LiDAR-derived Digital Surface Model (DSM): A 1m resolution DSM of Cardiff, UK, used as a ground truth for comparison.
  • Oblique Airborne Photogrammetry-based 3D city model (OAP3D): A high-resolution OAP3D of Weihai, China, used as a ground truth for comparison.

The paper does not introduce a new benchmark dataset but rather leverages existing street view imagery and compares its results against established high-resolution datasets.

Key Insights and Ideas

  • Street View as a Rich Data Source for SVF: The paper demonstrates that readily available street view imagery, often collected for navigation and mapping, can be repurposed for detailed environmental analysis like SVF estimation. This opens up vast possibilities for urban studies.
  • Deep Learning for Robust Sky Delineation: Using SegNet for sky segmentation proved effective, achieving high accuracy in classifying sky pixels, which is a critical and often labor-intensive step in traditional SVF methods.
  • Scalability for Big Data: The proposed method, particularly with GPU acceleration, shows excellent computational performance, making it suitable for processing the massive volumes of data generated by street view services.
  • Cost-Effectiveness: By avoiding the need for expensive LiDAR or aerial surveys, this approach offers a significantly more affordable way to obtain spatially continuous SVF data.

Key Results

  • Comparison with LiDAR-DSM: SVF estimates from GSV showed a correlation coefficient (R) of 0.863 with a LiDAR-derived DSM. The accuracy was assessed with an average of 92% sky pixel identification by SegNet.
  • Comparison with OAP3D: SVF estimates from TSV demonstrated a very high correlation coefficient (R) of 0.987 with an oblique airborne photogrammetry-based 3D city model.
  • Manhattan Application: An analysis of approximately 11,709 GSV panoramas in Manhattan revealed a spatial pattern of SVF, with lower values in the densely built southern tip and higher values in the north, correlating with building density.
  • Computational Performance: Using a GPU, the SegNet model could classify 11,709 panoramas in 3 hours, a significant speedup compared to CPU processing (30 hours) or previous SVM-based methods.

Limitations and Future Work

  • GPS Positioning Errors: The paper acknowledges that inaccuracies in street view GPS coordinates could affect the accuracy of SVF estimation, especially when comparing with precise DSM data. Future work should quantitatively evaluate the influence of these errors.
  • Vegetation Influence: The presence of tree canopies, which are sometimes poorly captured in DSMs or can obscure the sky in street view images, was found to influence discrepancies between GSV and DSM estimates. Filtering out samples with high vegetation index improved correlation.
  • Temporal Variations: Street view imagery is updated at varying frequencies, and the date of acquisition is not always available. This temporal mismatch with datasets like the Cardiff DSM (collected over several years) can contribute to discrepancies. The dynamic nature of tree canopies also needs consideration.
  • Alleyways and Lanes: Small, narrow urban features might not be well represented in street view imagery, potentially affecting area-averaged SVF calculations.
  • Further Development Needed: The authors emphasize that while the method shows great potential, further development is required before it can be fully deployed for urban climate and planning communities. This includes improving SegNet's robustness and conducting more direct comparisons with hemispherical photography.

Summary updated

Abstract

Hemispherical (fisheye) photography is a well-established approach for estimating the sky view factor (SVF). High-resolution urban models from LiDAR and oblique airborne photogrammetry can provide continuous SVF estimates over a large urban area, but such data are not always available and are difficult to acquire. Street view panoramas have become widely available in urban areas worldwide: Google Street View (GSV) maintains a global network of panoramas excluding China and several other countries; Baidu Street View (BSV) and Tencent Street View (TSV) focus their panorama acquisition efforts within China, and have covered hundreds of cities therein. In this paper, we approach this issue from a big data perspective by presenting and validating a method for automatic estimation of SVF from massive amounts of street view photographs. Comparisons were made with SVF estimates derived from two independent sources: a LiDAR-based Digital Surface Model (DSM) and an oblique airborne photogrammetry-based 3D city model (OAP3D), resulting in a correlation coefficient of 0.863 and 0.987, respectively. The comparisons demonstrated the capacity of the proposed method to provide reliable SVF estimates. Additionally, we present an application of the proposed method with about 12,000 GSV panoramas to characterize the spatial distribution of SVF over Manhattan Island in New York City. Although this is a proof-of-concept study, it has shown the potential of the proposed approach to assist urban climate and urban planning research. However, further development is needed before this approach can be finally delivered to the urban climate and urban planning communities for practical applications.
BibTeX
@inproceedings{liang2017automatic,
  author = {Jianming Liang and Jianhua Gong and Jun Sun and Jieping Zhou and Wenhang Li and Yi Li and Liu Jin and Shen Shen},
  title = {{Automatic Sky View Factor Estimation from Street View Photographs—A Big Data Approach}},
  booktitle = {Remote},
  year = {2017},
  doi = {10.3390/rs9050411},
}
Figure