A training-free method for estimating the relative height of buildings

Zhe Chen, Chengjie Li, Fuxun Liang, P. S. Tong, Long Chen, Zhen Dong, Bisheng Yang2025 · DOI

TL;DR

Researchers have developed a "training-free" pipeline that uses the Depth Anything V2 foundation model to estimate individual building heights from single, high-resolution satellite images. By combining zero-shot depth estimation with a novel graph-based optimization and a viewpoint bias filter, the method achieves high-fidelity 3D urban mapping without requiring expensive LiDAR or massive labeled datasets.

What Problem Does This Paper Address?

As cities grow vertically, understanding 3D urban morphology is critical for energy modeling, disaster planning, and carbon emission tracking. However, getting accurate building heights at scale is surprisingly difficult.

Traditional methods like Airborne LiDAR provide excellent precision but are prohibitively expensive for large-scale or frequent updates. Stereo photogrammetry (using multiple satellite views) is complex and often suffers from occlusions in dense "urban canyons." Meanwhile, existing deep learning models for monocular (single-image) height estimation usually require vast amounts of annotated training data and often fail to generalize when applied to cities with different architectural styles or taller buildings.

The authors aim to bridge this gap by creating a scalable, high-resolution method that works "out of the box" on standard satellite imagery.

Methods

The proposed framework consists of three main stages:

  1. Patch-based Relative Depth Estimation: To handle the massive size of remote sensing (RS) images while staying within GPU memory limits, the system crops imagery into 1024x1024 patches with a 75% overlap. It then runs Depth Anything V2 (DAv2)—a state-of-the-art computer vision foundation model—on each patch to generate relative depth maps.
  2. Height-Weighted Graph Optimization (HWGO): Because DAv2 estimates depth independently for each patch, the resulting maps have inconsistent scales and offsets. The authors treat the patches as nodes in a graph and use the overlapping regions to align them. Crucially, they introduce a height-weighted error function that prioritizes alignment accuracy for pixels with significant height differences (over 10m), ensuring that buildings—the primary interest—are aligned more precisely than flat ground. They solve this using the Levenberg–Marquardt algorithm.
  3. View Bias Filter (VBF): Satellite images are perspective views, meaning the "depth" provided by the model includes the distance to the sensor and the tilt of the viewpoint. To convert this into "relative height" (distance from the ground), the authors apply a morphological filter (dilation and erosion) to estimate the underlying terrain/viewpoint bias and subtract it from the depth map.

Datasets

The method was evaluated using:

  • Input Data: Google Earth satellite imagery with 0.5-meter resolution covering a 20 $km^2$ area in the Optics Valley of Wuhan, China.
  • Validation Data: Airborne LiDAR data (processed into a normalized Digital Surface Model, or nDSM) collected in 2020 served as the ground truth for accuracy assessment.

Key Insights and Ideas

  • Foundation Models are Ready for Space: The paper demonstrates that foundation models trained on natural images (like DAv2) possess enough "geometric common sense" to be highly effective for remote sensing tasks without any domain-specific fine-tuning.
  • Prioritizing Buildings in Optimization: A key "aha moment" is the weighting strategy in the graph optimization. By essentially telling the algorithm "care more about the tall stuff when stitching these patches together," they significantly reduced the artifacts and "seams" typically found in patch-based depth estimation.
  • Morphology as a Proxy for DEM: Instead of requiring a separate Digital Elevation Model (DEM), the authors used morphological operations to "flatten" the perspective bias. This makes the pipeline truly self-contained and applicable even in regions where high-res elevation data is unavailable.

Key Results

The method significantly outperformed the previous state-of-the-art (SOTA) supervised model, MLSBRN:

  • Accuracy: The proposed method achieved an $R^2$ of 0.73, compared to just 0.25 for MLSBRN.
  • Error Reduction: The Relative Root Mean Square Error (RRMSE) was slashed from 0.53 (SOTA) to 0.25 (Ours).
  • Qualitative Performance: Unlike previous models that tended to "smooth out" or underestimate the height of skyscrapers, this method maintained sharp boundaries and accurately captured the heights of tall structures.

Limitations and Future Work

While the method is a major step forward for zero-shot 3D mapping, the authors note a few areas for improvement:

  • Relative vs. Absolute: The current method provides relative building heights. To get absolute heights (meters above sea level), the system would need to integrate external reference data.
  • Imaging Conditions: The model's performance can still be affected by extreme sun angles, heavy shadows, or atmospheric conditions that obscure building footprints.
  • Adaptability: Future work will focus on making the method more robust to varying sensor types and imaging geometries to ensure consistent performance across different satellite constellations.

Summary updated

Abstract

Urbanization’s vertical shift underscores the need for accurate building height estimation to support sustainable planning. Existing methods, limited by low-resolution data and poor generalization, cannot resolve individual buildings. We propose a training-free approach leveraging the foundation model Depth Anything V2 for relative depth estimation from high-resolution remote sensing (RS) imagery. To address GPU memory constraints, RS images are cropped into overlapping patches, and their depth predictions are unified using a height-weighted graph optimization with Levenberg–Marquardt refinement, prioritizing building-related errors. A viewpoint bias filter, modeled as terrain variation, converts relative depth to height by subtracting a morphologically derived DEM. Experiments on Google satellite imagery with 0.5 m resolution over 20 km2 in Wuhan, validated against airborne LiDAR, show an R2 of 0.73 for building heights, significantly outperforming the state-of-the-art MLSBRN whose R2 is only 0.25 and which underestimates tall buildings. Without annotation or training, our scalable method accurately estimates individual building heights, generalizes across complex urban morphologies, and provides a robust solution for 3D urban studies.
BibTeX
@inproceedings{chen2025training,
  author = {Zhe Chen and Chengjie Li and Fuxun Liang and P. S. Tong and Long Chen and Zhen Dong and Bisheng Yang},
  title = {{A training-free method for estimating the relative height of buildings}},
  booktitle = {˜The},
  year = {2025},
  doi = {10.5194/isprs-archives-xlviii-1-w5-2025-7-2025},
}
Figure