Cloud photogrammetry with dense stereo for fisheye cameras

Christoph Beekmans, Johannes Schneider, Thomas Läbe, Martin Lennefer, Cyrill Stachniss, Clemens Simmer

Abstract

We present a novel approach for dense 3-D cloud reconstruction above an area of 10 × 10 km² using two hemispheric sky imagers with fisheye lenses in a stereo setup. We examine an epipolar rectification model designed for fisheye cameras, which allows the use of efficient out-of-the-box dense matching algorithms designed for classical pinhole-type cameras to search for correspondence information at every pixel. The resulting dense point cloud allows to recover a detailed and more complete cloud morphology compared to previous approaches that employed sparse feature-based stereo or assumed geometric constraints on the cloud field. Our approach is very efficient and can be fully automated. From the obtained 3-D shapes, cloud dynamics, size, motion, type and spacing can be derived, and used for radiation closure under cloudy conditions, for example. We implement and evaluate the proposed approach using real-world data and present two case studies: a stratocumulus layer validated against cloud radar and lidar-ceilometer data, and a rapid cumulus evolution in the presence of strong wind shear.

Introduction

Understanding cloud dynamics is critical for climate modeling and weather forecasting, yet clouds are notoriously difficult to measure in three dimensions. Traditional remote sensing tools have significant trade-offs. Cloud radars and lidars provide high-accuracy data but only along a narrow "pencil beam," making it difficult to capture the full morphology of fast-changing clouds. Satellite imaging offers broad coverage but lacks the vertical resolution and temporal frequency needed for convective cloud studies. Standard sky imagers usually only provide 2D information, such as cloud cover percentage.

Previous attempts at stereo cloud imaging often relied on sparse feature matching or assumed the clouds were flat. This research tackles the challenge of dense reconstruction—finding a 3D coordinate for nearly every pixel in a hemispheric (180°) view—using fisheye lenses that do not follow standard pinhole camera geometry.

Methodology

The proposed pipeline consists of four main stages to transform 2D hemispheric images into 3D data:

  1. Calibration: Accurate 3D data requires knowing the lens distortion (Interior Orientation) and the cameras' pointing direction (Exterior Orientation). The researchers used a specialized calibration cube for the lenses and utilized the positions of stars in the night sky to determine precise global orientation down to 0.1°.
  2. Epipolar Rectification: This is the core technical innovation. In standard stereo, "rectifying" images makes searching for matching points easy because a point in the left image will always be on the same horizontal row in the right image. The authors implemented a mathematical model to map non-linear fisheye "epipolar curves" onto straight image rows.
  3. Dense Matching: Once rectified, the Semi-Global Block Matching (SGBM) algorithm—a standard, efficient algorithm designed for pinhole cameras—is used to find correspondences for every pixel, even in low-contrast cloud regions.
  4. Triangulation: The matched pixels are projected back into 3D space to create a dense point cloud.
Overview of the system: (left) the sky imager hardware, (center) a raw image of a cumulus cloud, and (right) the derived 3D height map.

Key Results and Insights

The system was tested at the JOYCE (Jülich Observatory for Cloud Evolution) site in Germany. Key findings include:

  • High Accuracy: For stratocumulus clouds, the mean cloud-base height derived from the cameras was within 1% to 5% of the heights measured by a lidar-ceilometer.
  • Morphological Detail: The system successfully captured the "concave" and "skewed" shapes of cumulus turrets, which are typically invisible to zenith-pointing instruments.
  • Coverage: The setup provided detailed 3D geometries over an area of approximately 10 × 10 km² around the cameras.
  • Fisheye to Pinhole Bridge: The research demonstrates that specialized rectification allows the use of optimized, "off-the-shelf" computer vision research designed for standard cameras, bypassing the need to reinvent stereo matching for fisheye lenses.

Limitations and Future Work

While successful, the method has specific requirements. Clock synchronization is critical; even a 3-second offset between camera shutters can lead to significant height errors. Furthermore, the current 300m baseline between cameras is optimized for boundary layer clouds; higher-altitude clouds like cirrus would require a much larger distance to maintain triangulation accuracy. Future work will focus on adding a third camera to reduce occlusions and integrating these 3D morphologies with radiative transfer models to better understand cloud effects on the Earth's energy balance.

Summary updated

Abstract

We present a novel approach for dense 3-D cloud reconstruction above an area of 10 × 10 km2 using two hemispheric sky imagers with fisheye lenses in a stereo setup. We examine an epipolar rectification model designed for fisheye cameras, which allows the use of efficient out-of-the-box dense matching algorithms designed for classical pinhole-type cameras to search for correspondence information at every pixel. The resulting dense point cloud allows to recover a detailed and more complete cloud morphology compared to previous approaches that employed sparse feature-based stereo or assumed geometric constraints on the cloud field. Our approach is very efficient and can be fully automated. From the obtained 3-D shapes, cloud dynamics, size, motion, type and spacing can be derived, and used for radiation closure under cloudy conditions, for example. Fisheye lenses follow a different projection function than classical pinhole-type cameras and provide a large field of view with a single image. However, the computation of dense 3-D information is more complicated and standard implementations for dense 3-D stereo reconstruction cannot be easily applied. Together with an appropriate camera calibration, which includes internal camera geometry, global position and orientation of the stereo camera pair, we use the correspondence information from the stereo matching for dense 3-D stereo reconstruction of clouds located around the cameras. We implement and evaluate the proposed approach using real world data and present two case studies. In the first case, we validate the quality and accuracy of the method by comparing the stereo reconstruction of a stratocumulus layer with reflectivity observations measured by a cloud radar and the cloud-base height estimated from a Lidar-ceilometer. The second case analyzes a rapid cumulus evolution in the presence of strong wind shear.
BibTeX
@inproceedings{beekmans2016cloud,
  author = {Christoph Beekmans and Johannes Schneider and Thomas Läbe and Martin Lennefer and Cyrill Stachniss and Clemens Simmer},
  title = {{Cloud photogrammetry with dense stereo for fisheye cameras}},
  booktitle = {Atmospheric},
  year = {2016},
  doi = {10.5194/acp-16-14231-2016},
}
Figure