Building Morphology as a Shared Anchor for Multilingual, Multi-View Urban Vision-Language Models

Vision-language models describe aerial images of cities worse in Chinese than in English. They also degrade when the camera tilts away from straight-down. These look like two separate problems, and the literature treats them separately. We argue they share one cause. The model's internal picture of urban form is tied to surface cues, such as language-specific phrasing and view-specific pixel patterns, rather than to the physical quantities that actually distinguish one city block from another. Those quantities are building coverage, building height, facade area relative to ground area, and how much sky is visible from the street. They are measurable, they do not depend on language or camera angle, and urban climate research has shown that they drive surface heating.

The proposal is to train a multilingual image-text model with one extra constraint. Every embedding, whether it comes from a nadir image, an oblique image, an English caption or a Chinese caption, must predict the same measured building-morphology vector through a single shared linear readout. If the shared readout is the right anchor, it should shrink the language gap and the viewpoint gap at once. It should also yield a representation whose implied heat drivers agree in sign with what regression studies of land surface temperature have found. The test is whether one physical target can do the work that currently needs two separate fixes.

Core Claim

The claim is that a physically defined, language-agnostic and view-agnostic target makes cross-lingual and cross-view alignment better than contrastive image-text matching alone. In this setting the target is the block-level morphology vector $m$: building coverage ratio, mean building height, ground-to-facade area ratio, and sky view factor, each standardized.

Concretely, let $f_I$ be the image encoder and $f_T$ the multilingual text encoder. We add a regression term to the usual contrastive loss $\mathcal{L}_{\text{con}}$:

$$ \mathcal{L} = \mathcal{L}_{\text{con}} + \lambda \sum_{v} \lVert W f_I(x_v) - m \rVert^2 + \lambda \sum_{\ell} \lVert W f_T(t_\ell) - m \rVert^2 $$

In this objective, $x_v$ is the image of a block from viewpoint $v$ and $t_\ell$ is its caption in language $\ell$. The weight $\lambda$ sets the strength of the regression terms. $W$ is a single linear map shared by all views and all languages, and that sharing is the key ingredient. The term is cheap and easy to ablate. Its prediction is falsifiable: if the language gap and the viewpoint gap are caused by representations latching onto surface cues, forcing every embedding through the same physical readout should reduce both gaps more than augmentation or translation does.

Why These Three Directions Meet Here

The urban heat literature supplies the anchor and the physics check. Three studies provide the evidence. Variance partitioning over Shanghai found that 2D building metrics uniquely explain 51.0% of surface-temperature variance and 3D metrics a further 10.2%, with $R^2 = 0.785$ under a spatial error model (Quantifying the Relationship between 2D/3D Building Patterns and Land Surface Temperature). A three-megacity study reported building coverage positively correlated with surface temperature (mean $\rho = 0.486$) and building height negatively correlated (mean $\rho = -0.345$) (Exploring the Impact of 2-D/3-D Building Morphology on the Land Surface Temperature). A Beijing study showed that height cools mainly in summer daytime (Relationships between Land Surface Temperatures and Neighboring Environment). Together these tell us which morphology variables matter and what sign each should carry, so they define $m$ and give a downstream sanity check.

The same literature also constrains us. Relationships shift by functional zone (Do Urban Functional Zones Affect Land Surface Temperature Differently?) and by climate (Exploring the Seasonal Comparison of Land Surface Temperature Dominant Factors in the Tibetan Plateau). We therefore check signs within zones rather than claiming one city-wide law.

The viewpoint literature supplies the problem and the strongest view-side competitor. Discrete latent perspective learning, which synthesizes virtual viewpoints by homography in feature space, beats pixel augmentation on drone segmentation (Discrete Latent Perspective Learning for Segmentation and Detection). It relies on keypoint detectors, and the authors note that these fail on low-texture scenes, which describes many flat rooftops. CLIP-GS aligns 3D Gaussian scene representations with text by voting over rendered views (CLIP-GS). It works on isolated synthetic objects with English text only. Its multi-view voting is a cousin of our shared readout, but its anchor is still an image embedding, not a physical quantity. Prior work also motivates the view axis by showing that views of different objects from the same angle can look more alike than two views of the same object (Exploiting View-Specific Appearance Similarities Across Classes). The VisDrone challenge documents how badly ground-trained models fail under oblique, dense urban views (Vision Meets Drones).

The cross-lingual literature supplies the base models and the language gap. AltCLIP swaps CLIP's text encoder for a multilingual one by distillation (AltCLIP). MURAL regularizes with translation pairs (MURAL). CCLM treats captions and translations as views of one concept and was the first to beat translate-test (Cross-View Language Modeling). All three align language to images or to other languages; none aligns to a measured physical variable.

BRSIC is the closest prior work. It added expert Chinese captions to three nadir remote-sensing caption sets and found a persistent English advantage even after fine-tuning (A Benchmark for Multi-Lingual Vision-Language Learning in Remote Sensing Image Captioning). It uses nadir images only and attaches no physical targets. Our work differs on both counts.

Research Program

The first component is data. We attach morphology vectors to BRSIC's images by geolocating tiles against GLAMOUR's global height and footprint grids (GLAMOUR). Where finer 3D building data exist, as in the Chinese megacity studies, we compute the ground-to-facade ratio and sky view factor directly.

GLAMOUR's labels are noisy (height RMSE 7.5 m, Pearson correlation 0.54), so we treat them as weak targets and report results against the cleaner 3D building data separately. For the view axis, we build paired nadir and oblique image sets over the same blocks. We render oblique views from 3D city models and supplement them with real oblique aerial or drone imagery where coverage exists. We collect English and Chinese captions for a held-out subset, plus a small evaluation set in a third, lower-resource language. This construction step is the main cost and the main risk, and we budget for it explicitly.

The second component is the model. We start from AltCLIP's multilingual encoder pair and fine-tune on the mixed-view, mixed-language data with the objective above. The readout $W$ is shared across all views and languages. We sweep $\lambda$ and the morphology subset: 2D only, 3D only, and both.

The third component is analysis. We measure the viewpoint gap as the drop in retrieval and morphology-regression accuracy from nadir to oblique, and the language gap as the drop from English to Chinese and to the third language. We then probe the predicted morphology for physical consistency. Within each functional zone, partial correlations between embedding-predicted morphology and Landsat-derived surface temperature should reproduce the literature's signs: coverage positive, height negative in summer daytime, sky view factor negative.

Baselines

The naive reference is AltCLIP used zero-shot, with a linear morphology probe fit on English nadir embeddings and then applied unchanged to oblique images and non-English text. This measures how large the two gaps are before any intervention. BRSIC reports how poor zero-shot Chinese can be in this domain: Qwen2-VL-7B reaches a Chinese CIDEr of only 15.95 on Sydney-Captions zero-shot (A Benchmark for Multi-Lingual Vision-Language Learning in Remote Sensing Image Captioning).

The strongest existing language-side competitor is supervised fine-tuning on BRSIC's bilingual captions. BRSIC reports Chinese CIDEr 251.52 on Sydney-Captions for Qwen2-VL-7B after fine-tuning. That is a captioning number. For the retrieval and morphology-regression tasks we evaluate, there is no reported result on Lacuna for BRSIC, so we run this baseline ourselves. We fine-tune the same AltCLIP backbone contrastively on the same bilingual and multi-view data, using the same steps, compute and hyperparameter search budget, and omit only the morphology term.

The strongest view-side competitor is DLPL, which reports a +5.1% mIoU gain over SegFormer on DroneSeg and a 6.6% versus 3.7% gain over AutoAugment (Discrete Latent Perspective Learning for Segmentation and Detection). We apply its latent homography augmentation to the image encoder under the identical budget. We also report the combination of DLPL and fine-tuning, which is the fair "two separate fixes" competitor.

Beating these baselines means two things. Our model must narrow both gaps more than the combined baseline, while keeping English nadir performance within noise. It must also do so on the third language, which none of the methods saw paired with morphology.

The ablation that isolates the key ingredient replaces the shared $W$ with separate readouts per view and per language. A second control trains with shuffled morphology targets, which rules out an effect that comes from regularization alone. If separate readouts work as well as the shared one, the anchoring claim fails.

Why not just…?

Why not just translate the query to English and use an English remote-sensing model? For Chinese this is a real threat. Machine translation is strong, and CCLM is the only cited work shown to beat translate-test (Cross-View Language Modeling), on general web images. BRSIC found that machine translation garbles domain terms such as "cloverleaf interchange," but that is anecdotal. We run translate-test with the best available translation model as a full baseline in every language. If it matches us in Chinese, we will say so and rest the cross-lingual claim on the lower-resource language, where translation quality is weaker.

Why not just ask a language model to turn the description into numeric thresholds and look the answer up in GLAMOUR? For structured queries over places GLAMOUR covers well, this would work, and it needs no vision model. It fails where the rasters are wrong or too coarse. GLAMOUR's height estimates correlate only 0.54 with reference data. It also cannot handle a new oblique photograph of a street. We test this lookup pipeline directly on our held-out blocks, measuring retrieval accuracy against the 3D building data. Our claim is limited to the image-conditioned case, where the lookup cannot act.

Why not just use stronger augmentation, or 3D Gaussian representations, for the viewpoint problem? DLPL is included as a baseline. CLIP-GS-style multi-view voting is close to our shared readout, so we add a variant that votes across rendered views without a morphology target. If voting alone closes the viewpoint gap, the physical anchor contributes nothing on the view axis, and we would report that. Keypoint failures on low-texture rooftops (Discrete Latent Perspective Learning for Segmentation and Detection) give a specific subset where we expect the physical target to help most. We report results on that subset separately.

Decisive Checks

The first check is the shared-versus-separate readout ablation. If a shared $W$ does not beat per-view and per-language heads on both gaps, the central claim is wrong.

The second check is third-language transfer. That language contributes captions only at test time, so any gap reduction there must come from the anchor rather than from paired supervision.

The third check is physical consistency. If embedding-predicted morphology reproduces the published signs of the temperature relationships within functional zones, and baseline embeddings do not, the representation is tracking the quantities that matter for urban heat rather than merely regressing to labels. If neither check separates our model from contrastive fine-tuning plus DLPL, the honest conclusion is that the two gaps have separate causes, and we would report that.