3D exploitation of large urban photo archives

Peter Cho, Noah Snavely, Ross AndersonProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2010 · DOI

TL;DR

This paper presents a system that fuses uncooperatively collected 2D urban photos with 3D aerial LiDAR and GIS data to create georegistered 3D urban maps. This allows for novel geometry-based photo enhancements like automatic building annotation and image-based querying, paving the way for augmented reality experiences on mobile devices.

What Problem Does This Paper Address?

The proliferation of readily available urban imagery from sources like Flickr presents a wealth of data, but it's largely unstructured and difficult to relate to one another or to external information. Traditional image processing struggles to extract meaningful, high-level knowledge from these unorganized collections. The authors aim to bridge this gap by leveraging 3D geometry as an organizing principle, enabling the transfer of absolute spatial and semantic information into 2D images, thereby making them more navigable and informative.

Methods

The core of the system involves several key steps:

  1. 3D Photo Reconstruction: Using the Structure from Motion (SfM) approach, the system first reconstructs the relative camera poses (orientation and position) and sparse 3D scene geometry from a large collection of uncooperatively captured photos. This involves extracting Scale-Invariant Feature Transform (SIFT) features, matching them across images, and then solving for camera parameters and 3D points.
  2. 3D Map Construction: This step fuses multiple data sources to create an absolute 3D urban map. It combines:
  3. Aerial LiDAR data: Provides a dense 3D point cloud of the urban environment with precise geolocations.
  4. Satellite imagery: Offers high-resolution overhead views to add color and texture.
  5. GIS layers: Incorporates semantic information like landmarks, transportation routes, and political zones, with their geocoordinates. The LiDAR data is used to infer altitude information for GIS points.

  6. Reconstructed Photo Georegistration: The reconstructed 3D photo geometry is then aligned with the absolute 3D urban map. This is achieved through a manual process of identifying corresponding features in both the SfM point cloud and the LiDAR map, followed by a least-squares fitting to determine the global transformation.

  7. Urban Knowledge Propagation: Once georegistered, high-level knowledge from the 3D map can be projected into the 2D images. Two proof-of-concept applications are demonstrated:
  8. Urban Feature Annotation: Building names and other geolocated information are projected onto the corresponding image planes. Line-of-sight occlusion is handled by assuming walls drop vertically from rooftop LiDAR data.
  9. Image-Based Querying: A system allows users to select pixels in a georegistered photo, which are then raytraced into the 3D map to retrieve their geocoordinates and range. This information can then be projected back into other images.

Datasets

  • Flickr Photos: Over 1000 uncooperatively collected photos of the lower Manhattan skyline and the Statue of Liberty were downloaded from Flickr.
  • Aerial LiDAR Data: A Rapid Terrain Visualization map of New York City collected on October 15, 2001, with a 1-meter ground sampling distance.
  • Satellite Imagery: Quickbird satellite imagery covering the same area as the LiDAR data, with a 0.8-meter ground sampling distance.
  • GIS Layers: Standard GIS data including points (landmarks), curves (routes), and regions (political zones).

The paper does not introduce a new benchmark dataset but rather uses existing data to demonstrate its system's capabilities.

Key Insights and Ideas

  • 3D Geometry as an Organizing Principle: The paper strongly advocates for using 3D scene geometry as the fundamental organizing principle for large, uncurated collections of urban imagery. This allows for a unified framework to integrate diverse data sources.
  • Bridging the Gap Between 2D and 3D: The system effectively bridges the gap between unstructured 2D photos and structured 3D urban environments, enabling the transfer of absolute spatial and semantic information.
  • Geometry-Driven Enhancements: Novel applications like automatic building annotation and image-based querying are made possible by this geometric fusion, which are difficult or impossible with conventional 2D image processing alone.
  • Potential for Augmented Reality: The work lays the groundwork for real-time augmented reality experiences on mobile devices, where contextual information can be overlaid onto live camera feeds.

Key Results

  • Successful Georegistration: The system successfully georegisters over 1000 reconstructed photos with the 3D NYC map.
  • Accurate Alignment: An average angular orientation error of approximately 0.1 degrees is estimated for the georegistered cameras of the Statue of Liberty.
  • Qualitative Alignment: Visual inspection shows good qualitative alignment between skyline photos and the 3D map, demonstrating the system's ability to integrate independent data sets.
  • Proof-of-Concept Demonstrations: The paper successfully demonstrates automatic building annotation and an image-based querying tool, showcasing the practical utility of the developed system.

Limitations and Future Work

Limitations:

  • Manual Georegistration Step: The georegistration of the SfM reconstruction to the absolute 3D map currently requires a manual step, which the authors acknowledge could be automated.
  • Static Occluders: The urban feature annotation approach only handles static occluders (buildings) and not transient ones like people or cars.
  • Computational Complexity: 3D reconstruction for large numbers of photos is computationally intensive and requires parallel processing.

Future Work:

  • Automated Georegistration: Automating the manual georegistration step, potentially by incorporating GPS information or using more advanced alignment techniques.
  • Real-time Interpretation: Developing real-time interpretation of pixel outputs from mobile cameras for instantaneous image enhancement and querying.
  • Mobile Exploitation: Extending the algorithms to handle larger datasets and more complex urban environments (e.g., MIT campus with significant urban canyons) for mobile exploitation.
  • Dynamic Occluders: Addressing the challenge of transient occluders for more robust annotation.
  • Non-rigid Deformations: Investigating methods to correct for non-rigid deformations between different data sources.

Summary updated

Abstract

Recent work in computer vision has demonstrated the potential to automatically recover camera and scene geometry from large collections of uncooperatively-collected photos. At the same time, aerial ladar and Geographic Information System (GIS) data are becoming more readily accessible. In this paper, we present a system for fusing these data sources in order to transfer 3D and GIS information into outdoor urban imagery. Applying this system to 1000+ pictures shot of the lower Manhattan skyline and the Statue of Liberty, we present two proof-of-concept examples of geometry-based photo enhancement which are difficult to perform via conventional image processing: feature annotation and image-based querying. In these examples, high-level knowledge projects from 3D world-space into georegistered 2D image planes and/or propagates between different photos. Such automatic capabilities lay the groundwork for future real-time labeling of imagery shot in complex city environments by mobile smart phones.
BibTeX
@inproceedings{cho2010d,
  author = {Peter Cho and Noah Snavely and Ross Anderson},
  title = {{3D exploitation of large urban photo archives}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2010},
  doi = {10.1117/12.849767},
}
Figure