Estimating how tall a building is from a single overhead satellite image is mostly a matter of looking in the right place. A tall tower's shadow can land tens to hundreds of meters away, across a street or on top of a neighbouring roof. In off-nadir images, where the satellite looks from an angle rather than straight down, the tower's facade also leans away from its footprint. Both offsets point in directions that are known before the network sees a pixel: the sun's azimuth and elevation, and the satellite's viewing azimuth and angle, both normally recorded with the acquisition. Height networks today ignore this. Convolutional encoders see only a small neighbourhood. Transformers see everything, pay a quadratic cost, and must tile the image. Neither is told which direction holds the evidence.
We propose to give a height network a long, thin receptive field that runs along those two physical directions. At a few encoder stages, each pixel reads one-dimensionally along the line toward its own shadow and along the line of facade lean. The cost is linear in image size times line length. The model is trained with a differentiable shadow-consistency loss that renders the hard shadow implied by the predicted heights and compares it with the observed dark regions. The claim is narrow and testable. Most of the benefit of "global context" for building height is supplied by a directional, physically aligned context. Getting the direction right should matter more than getting the range large, and it should matter most for the tall buildings where global height products currently fail.
Core Claim and What Is New
The closest prior work falls into three groups, and each has one piece of this idea but not the combination.
The first group supplies the motivation. TEMPO uses a convolutional EfficientNet-B6 U-Net and reports that it struggles with extreme vertical outliers; its authors name explicit shadow use as future work. GLAMOUR underestimates height in dense high-rise cores using fixed-patch multi-task networks.
The second group addresses non-local context but without physics. Disentangled Latent Transformer replaces the CNN with dense patch attention and roughly halves scale-invariant error on synthetic GTAH relative to CNN baselines. That attention is isotropic, expensive, and has no notion of sun or view direction. Deep ordinal regression enlarges context with fixed-rate dilated pooling (ASPP), which is radially symmetric, and it still works on 256×256 patches.
The third group has the right physics or the right operator, but in another setting. Controllable Shadow Generation Using Pixel Height Maps couples predicted height to a differentiable hard-shadow renderer, but only for ground-level object cutouts. CCNet shows that attention along rows and columns recovers dense context at a fraction of the memory. Butterworth as Attention shows that learned anisotropic filters capture directional urban texture that isotropic transformers blur, but it learns the directions from data and applies them to pansharpening.
Our proposal takes CCNet-style axial routing and replaces the arbitrary image axes with axes fixed per image by the acquisition geometry. It then takes the pixel-height shadow renderer and moves it to the overhead setting as a training signal.
Research Program
The core module is a geometry-rectified axial block. Given a feature map and the per-image sun azimuth $\phi_s$, it resamples the features into a frame where $\phi_s$ points along the column axis. It applies one-dimensional attention, or alternatively a ParCNetV2-style oversized 1D depthwise kernel (ParCNetV2), down each column. It then resamples back and adds the result residually to the unrotated features. A second branch does the same along the view azimuth $\phi_v$. The residual path matters. Rotating features means interpolating them, which blurs high frequencies, and the Butterworth paper shows that blurring is what melts building edges. The skip connection keeps the convolutional path's sharp boundaries intact.
We restrict the attention to the half-line on the shadow side, since a roof's shadow lies only away from the sun, and cap its length using the solar elevation. The shadow of a building of height $h$ has ground length $h/\tan\theta_s$, where $\theta_s$ is the solar elevation. A cap set from the tallest plausible building in the region therefore bounds the line length, and with it the cost.
The training signal adds a shadow-consistency term to the standard height loss. A differentiable hard-shadow renderer $R$ projects the predicted height map $\hat h$ along $(\phi_s,\theta_s)$ to produce a soft shadow mask. The loss is
$$\mathcal{L} = \mathcal{L}_{\text{height}}(\hat h, h) + \lambda\,\mathrm{BCE}\big(R(\hat h;\phi_s,\theta_s),\, S\big),$$
where $h$ is the reference normalized surface model (nDSM), $S$ is a shadow mask from a pretrained shadow detector, and $\lambda$ weights the term. This loss can also be applied to imagery that has no height labels, which is how we test whether it helps transfer across cities. When an image lacks sun metadata, we estimate $\phi_s$ with a small regressor trained on images where it is known. We report results with recorded angles and with estimated angles separately.
Baselines
The naive reference point is a multi-task U-Net with an EfficientNet backbone in the style of TEMPO, trained with plain height regression. It shows how much any context mechanism adds over local convolution.
The strongest published monocular baseline in the file is the ordinal-regression network with ASPP of Xiang Li et al.. It reports RMSE 1.404 m on ISPRS Potsdam and 1.698 m on ISPRS Vaihingen. We will both quote these figures and retrain the network under our splits.
On DFC23 Track 2, the winning HGDNet ensemble from the DFC23 outcome paper reports 45.0% mAP for joint building extraction and height. Our module will be inserted into a single HGDNet-style height-hierarchy model and compared against that single model, not against the ensemble.
As a dense-attention competitor we train a DLT-style transformer (Disentangled Latent Transformer). The file reports only a relative gain on GTAH; no reported absolute height RMSE for it on DFC2019 was found on Lacuna.
The key ablation is unaligned criss-cross attention (CCNet) inserted at the same stages with matched parameters and FLOPs. It isolates direction from range. A second ablation feeds deliberately wrong angles: a random azimuth, or azimuth plus 180°.
All models share the same backbone, input resolution, tile size, training epochs, and a fixed hyperparameter search budget of equal size per method, run with three seeds. Beating the field means two things. First, lower overall RMSE than the retrained ASPP model and the transformer at equal or lower compute. Second, and more important, a clearly larger reduction in error on buildings above roughly 30 m, the stratum where TEMPO and GLAMOUR report failure. A gain only in the overall average would not count.
Why not just…?
Why not just give the CNN the sun and view angles as extra input channels? This conditioning, in the spirit of coordinate channels and global broadcasting (Broadcasting Convolutional Network), tells the network which direction to look but does not let a pixel reach a shadow 80 m away. A local kernel still has to relay the information through many layers. We train exactly this variant at matched compute. If it matches our module on tall buildings, our claim that the receptive field shape is the bottleneck is wrong.
Why not just rotate the input image so the sun points "down" and use a standard CNN with tall vertical kernels? This is the most serious threat, because it is nearly our method done at the input stage. It can align only one direction per image, so in off-nadir imagery it cannot also align facade lean. We will run it, using ParCNetV2 vertical kernels, as a baseline. If it equals the feature-level module on near-nadir data, we will report input rectification as the recommended practical method there. Our module would then be justified only by gains on off-nadir subsets.
Why not just use a global transformer such as Swin or DLT? Global attention can in principle find the shadow. It must learn the direction from data, costs quadratically, forces tiling, and smooths edges (Butterworth as Attention). The matched-compute comparison and a boundary-error metric, height error within 2 m of roof edges, will show whether the directional prior buys accuracy, efficiency, or neither.
Why not just widen ASPP dilations? ASPP samples a sparse, symmetric grid and leaves gaps between samples. The retrained ordinal baseline with enlarged rates directly tests whether range alone closes the gap.
Proposed Experiments
The main benchmarks are ISPRS Potsdam and Vaihingen (RMSE, MAE), DFC2019/US3D (RMSE and scale-invariant RMSE), and DFC23 Track 2 (mAP and height RMSE). For DFC23 we stratify by height bin and by city, using its 17 cities for leave-cities-out transfer. On that transfer split, the shadow loss is applied to unlabeled target-city imagery. Before committing to them, we must confirm that sun and view metadata exist for each benchmark; this is a real risk, which is why the estimated-angle path exists. As a secondary off-nadir check, SpaceNet 4 footprint segmentation tests the view-azimuth branch alone. The baseline there is the 0.743 F1 on very-off-nadir imagery reported for MetaACM (Improving Building Segmentation for Off-Nadir Satellite Imagery).
Three checks decide the question. The first feeds wrong sun angles at test time: if the model relies on shadows, error on tall buildings should rise sharply, while the unaligned CCNet ablation should be unaffected. The second plots error against shadow length in pixels: the advantage over local and isotropic baselines should grow with shadow length. The third compares wall-clock time and memory against the dense transformer at full-tile resolution without stitching. If error does not depend on the given angle and the gain does not grow with shadow length, the directional hypothesis is refuted, whatever the average RMSE shows.
References
- TEMPO: Global Temporal Building Density and Height Estimation from Satellite Imagery
- GLAMOUR: GLobAl building MOrphology dataset for URban hydroclimate modelling
- Disentangled Latent Transformer for Interpretable Monocular Height Estimation
- Height Estimation From Single Aerial Images Using a Deep Ordinal Regression Network
- Controllable Shadow Generation Using Pixel Height Maps
- CCNet: Criss-Cross Attention for Semantic Segmentation
- Butterworth as Attention: Anisotropic Spectral Gating for Pansharpening
- ParCNetV2: Oversized Kernel with Enhanced Attention
- Large-Scale Fine-Grained Building Classification and Height Estimation for Semantic Urban Reconstruction: Outcome of the 2023 IEEE GRSS Data Fusion Contest
- Broadcasting Convolutional Network for Visual Relational Reasoning
- Improving Building Segmentation for Off-Nadir Satellite Imagery