A Generative Model for True Orthorectification

Christoph Strecha, Luc Van Gool, Pascal Fua

Abstract: Orthographic images compose an efficient and economic way to represent aerial images. This kind of information allows to measure two-dimensional objects and relate these to Geographic Information Systems. This paper deals with the computation of a true ortho-graphic image given a set of overlapping perspective images. These are, together with the internal and external calibration the only input to our approach. These few requirements form a large advantage to systems where the digital surface model (DSM), e.g. provided by LIDAR data, is necessary. We used a Bayesian approach and define a generative model of the input images. In this, the input images are regarded as noisy measurements of an underlying true and hence unknown orthoimage. These measurements are obtained by an image formation process (generative model) that involves apart from the true orthoimage several additional parameters. Our goal is to invert the image formation process by estimating those parameters which make our input images most likely. We present results on aerial images of a complex urban environment.

Introduction

Orthorectification is a fundamental process in photogrammetry and remote sensing, transforming perspective aerial photographs into map-like images where every pixel appears to be viewed from directly above. These "orthoimages" are essential for Geographic Information Systems (GIS) because they allow for direct distance and area measurements. However, standard orthorectification often fails in complex urban environments where tall structures "lean" over adjacent streets, creating occlusions and geometric inaccuracies.

Traditionally, achieving "true" orthorectification—where these occlusions are corrected—requires an external, high-resolution Digital Surface Model (DSM), typically acquired via expensive LIDAR scans. This paper introduces a generative Bayesian framework that eliminates the need for external DSMs by simultaneously estimating terrain depth and the orthoimage from overlapping perspective photos.

The Challenge of Perspective Distortion

Standard aerial images are captured through perspective projection. This leads to two primary issues: geometric distortion (building lean) and occlusions (buildings hiding the ground behind them). Furthermore, transient elements like moving vehicles, pedestrians, and specular sun reflections on glass surfaces introduce "noise" that should not appear on a static map.

Conceptual diagram of perspective distortion, building lean, and transient artifacts like sun glints.

As illustrated in the conceptual diagrams above, a true orthoimage must account for the 3D structure of the scene to correctly place every pixel in its geographic location, effectively "looking around" buildings using information from multiple overlapping views.

Methodology: A Generative Bayesian Framework

The core innovation of this research is treating the orthorectification process as an inverse problem. Instead of simply projecting pixels from source to destination, the authors propose a Generative Model. They ask: "What underlying 'ideal' orthoimage and terrain depth would most likely produce the set of perspective photos we actually captured?"

Technical Components

  1. Markov Random Field (MRF): The unobservable depth state of each pixel is modeled using an MRF. This enforces a "smoothness" prior, reflecting the physical reality that the world is generally composed of continuous surfaces rather than random depth spikes.
  2. Expectation-Maximization (EM) Algorithm: This iterative solver is used to find the Maximum Likelihood estimate of the model parameters.
  3. E-step: Estimates the expected depth and visibility (determining which pixels are occluded in which cameras) using Belief Propagation.
  4. M-step: Updates the parameters of the "ideal" orthoimage, noise levels, and global color transformations to account for lighting differences between camera passes.
  5. Outlier Modeling: The system explicitly classifies pixels as either "inliers" (consistent with the static terrain) or "outliers" (moving objects or photometric artifacts). This allows the model to statistically "ignore" transient features.

Experimental Results

The algorithm was tested on a high-resolution dataset of Zwolle, Netherlands, consisting of 68 images (11,500 x 7,500 pixels each) captured by a Vexcel UltraCam-D digital frame camera. The images were taken along three flight lines with significant overlap.

The resulting estimated Digital Surface Model (left) and the final true orthoimage (right) generated by the model.

As shown in the results, the model successfully produces a sharp, true orthoimage alongside a detailed depth map. A key advantage of this unified framework is its ability to handle uniform color regions (like large rooftops) better than traditional multi-view stereo methods. In areas where color is consistent and depth is ambiguous, the model relies on its MRF smoothness prior to maintain structural integrity.

Furthermore, the probabilistic approach naturally "cleans" the final map. Because moving cars appear in different locations across overlapping photos, they are statistically inconsistent with the static model and are automatically filtered out. Similarly, view-dependent specularities (sun glints) are downweighted, resulting in a final image free of distracting artifacts.

Conclusion and Future Work

This research demonstrates that high-quality true orthoimages can be generated without the need for external LIDAR data. By integrating depth estimation and image synthesis into a single probabilistic loop, the system achieves superior results in complex urban environments.

Future work aims to address the computational complexity of the visibility states, which currently grow combinatorially with the number of input images. While the current patch-based processing allows for parallelization, further optimization of the joint MRF for depth and visibility remains a primary goal for large-scale deployment.

Summary updated

Abstract

Orthographic images compose an efficient and economic way to represent aerial images. This kind of information allows to measure two-dimensional objects and relate these to Geographic Information Systems. This paper deals with the computation of a true ortho-graphic image given a set of overlapping perspective images. These are, together with the internal and external calibration the only input to our approach. These few requirements form a large advantage to systems where the digital surface model (DSM), e.g. provided by LIDAR data, is necessary. We used a Bayesian approach and define a generative model of the input images. In this, the input images are regarded as noisy measurements of an underlying true and hence unknown orthoimage. These measurements are obtained by an image formation process (generative model) that involves apart from the true orthoimage several additional parameters. Our goal is to invert the image formation process by estimating those parameters which make our input images most likely. We present results on aerial images of a complex urban environment.
BibTeX
@inproceedings{strecha2008generative,
  author = {Christoph Strecha and Luc Van Gool and Pascal Fua},
  title = {{A Generative Model for True Orthorectification}},
  booktitle = {Infoscience},
  year = {2008},
}
Figure