In very high-resolution aerial imagery, one class is really several materials. At 5 to 10 cm per pixel, a "building" is clay tiles, weathered concrete, glass skylights, solar panels and black chimney shadows. The shadow on that roof can look almost identical to the shadow on the road next to it. Vision-language models such as CLIP (an image encoder and a text encoder trained so that matching images and captions land near each other) now make it possible to segment such scenes from class names alone, with no labelled training data. Current methods still score every pixel against one text embedding per class. Pixels from atypical parts, such as shadows, skylights and stains, match none of the class names well, so they end up labelled by whichever class is least wrong.
This proposal treats a class as a mixture of nameable parts. Each pixel is softly assigned to a vocabulary of part descriptions such as "red terracotta roof tile", "dark cast shadow" or "glass skylight". Ambiguous parts like shadow are allowed to belong to no class. Each candidate object region is then labelled by how much of its mass falls on parts that belong to each class. The cross-modal angle is essential to the design. The step that assigns pixels to parts compares distances, and published analysis shows that distance-based steps are exactly where the "modality gap" does damage. That gap is the systematic offset separating image embeddings from text embeddings in CLIP's shared space. We therefore close the gap at the part-assignment step and leave it alone at the final ranking step.
Core Claim
The main claim is about where in a dense-prediction pipeline the modality gap matters. Closing the Modality Gap Aligns Group-Wise Semantics shows that cross-modal retrieval depends only on rank order and so barely notices the gap. Clustering depends on raw distances and degrades with the gap, which splits groups into parallel "semantic stripes." Mind the Gap shows the opposite side of the trade-off: keeping a non-zero gap can slightly improve zero-shot classification.
Open-vocabulary segmentation contains both kinds of step. The per-pixel choice of class is a ranking. Grouping heterogeneous pixels into objects, or assigning them to parts, is a clustering. We hypothesise two things. First, removing the gap helps only in the clustering step. Second, the benefit concentrates on the spectrally atypical pixels that cause most errors in high-resolution scenes. If both hold, the long-standing remote-sensing problem of intra-class spectral variance becomes a concrete, measurable instance of the gap-sensitivity theory. It also yields a training-free method that improves on single-prompt and averaged-prototype designs.
Mechanism
Let $x_p$ be the feature of image patch $p$ from a frozen CLIP-style encoder. We use the bias-removed, upsampled features of SegEarth-OV. Let $t_v$ be the text embedding of part description $v$ in a vocabulary $V$. Each part belongs either to one class's set $V_c$ or to a neutral set $V_0$ (shadow, specular glare, generic dark surface).
The simplest gap closure removes each modality's mean. We estimate $\mu_{\text{img}}$ over unlabelled Million-AID patches and $\mu_{\text{txt}}$ over the vocabulary, then renormalise, giving $\tilde x_p$ and $\tilde t_v$. The soft part assignment of patch $p$ is
$$a_p(v) = \frac{\exp(\cos(\tilde x_p, \tilde t_v)/\tau)}{\sum_{u \in V} \exp(\cos(\tilde x_p, \tilde t_u)/\tau)},$$
where $\tau$ is a temperature. For a class-agnostic region $R$, obtained by unsupervised over-segmentation of the upsampled features or from GIS footprints where available, the class score is the share of non-neutral part mass that falls on class $c$:
$$S_c(R) = \frac{\sum_{p \in R} \sum_{v \in V_c} a_p(v)}{\sum_{p \in R} \sum_{v \notin V_0} a_p(v)}.$$
The final pixel label combines $S_c(R)$ with the ordinary per-pixel score. That per-pixel score is computed on the uncentred features, because ranking is the gap-insensitive step.
The neutral set does the key work. A roof shadow sends its mass to "dark cast shadow" and so casts no vote. The surrounding tiles and concrete then decide the region's label. This is a cross-modal version of the finding by Wu et al. that roofs are characterised by distributions over heterogeneous colour and texture topics, not by one spectral signature. Here the topics are named in language rather than discovered, so the vocabulary is open. If plain mean-centering proves too crude, a secondary variant replaces it with a lightweight learned map trained across datasets. That map uses either LayerNorm-only tuning, following Tuning LayerNorm in Attention, or an Align-True-Pairs plus centroid-uniformity loss in the style of ModGap, applied only to the part-assignment branch.
Position Relative to Prior Work
Several papers motivate this proposal by documenting the variance problem. The urban classification studies of Huang et al. and Huang et al. show that pixel-wise classifiers fragment spectrally heterogeneous objects. PDSSNet and LOGCAN++ fight the same problem with ground-truth-guided prototypes and global and local class centres. Both are closed-set and supervised, and both reason only in visual feature space. We borrow their insight that local evidence must be aggregated against a class-level anchor. Our anchor is a text-defined part mixture, so new classes need no masks.
Three remote-sensing vision-language papers are the closest prior work, and each constrains the design. Embedding Generalized Semantic Knowledge into Few-Shot Remote Sensing Segmentation injects one class-description text embedding to stabilise high-variance classes. It conditions on a single monolithic text vector and never examines the gap. Bridging the Modality Gap via Visual Prototypes compensates for the gap with a one-dimensional affine rescaling of detection scores. It averages each class's diverse crops into one mean prototype, which is exactly the collapse of sub-components we avoid, and it handles only bounding boxes. SegEarth-OV and Open-Vocabulary Remote Sensing Image Semantic Segmentation handle global-token bias, resolution, rotation and scale. Neither models material heterogeneity or the gap, and SegEarth-OV reports remaining failures on spectrally fragmented roads and mixed land cover. We reuse SegEarth-OV's features as our substrate rather than competing with its feature pipeline.
On the gap side, the method's motivation rests on the duality between ranking and clustering in ModGap. Mitigating the Modality Gap with Fractal Spectral Geometry warns that global centroid shifts can distort local neighbourhoods. Our design addresses that warning in two ways: it applies centering only where distances are used, and it tests a neighbourhood-preserving learned variant. Cross the Gap shows that CLIP's image-to-image distances are less reliable than its image-to-text distances, including on EuroSAT. That is a direct argument against grouping pixels by visual affinity alone. Its per-query optimisation, which takes hundreds of steps, cannot scale to dense maps. Routing grouping through a fixed text vocabulary is our feed-forward substitute.
Research Program
The work has three parts. The first is diagnostic. On the ISPRS Potsdam and Vaihingen benchmarks we split building and impervious-surface pixels into lit and shadowed strata, using a brightness threshold, and on Potsdam a digital-surface-model-based shadow estimate. We then measure each patch's distance to the nearest text embedding. The prediction is that errors from training-free open-vocabulary methods concentrate on shadowed and atypical pixels, and that these pixels have the largest cross-modal distances. If the errors are not concentrated there, the premise fails and the proposal should stop.
The second part builds the part-voting segmenter. Part vocabularies are written once per class name by a fixed template procedure, with the neutral set held identical across datasets so that no benchmark-specific tuning leaks in.
The third part tests where the gap should be closed. We compare four placements of mean-centering, the learned variant, and no closure: none, at the part-assignment step only, at the per-pixel ranking step only, and at both.
Baselines
The do-nothing reference is SegEarth-OV run unchanged with one prompt per class. It is the strongest training-free open-vocabulary method in the file and also our feature substrate. The file reports only an average gain of 5.8% mIoU over eight datasets relative to earlier methods. No absolute per-dataset mIoU for SegEarth-OV on Potsdam, Vaihingen or LoveDA was found on Lacuna, so we rerun it under one shared protocol: same backbone, same upsampler, same sliding-window inference.
The strongest trained open-vocabulary competitor is the rotation-aggregation method of Open-Vocabulary Remote Sensing Image Semantic Segmentation. It reports 39.09% mIoU on iSAID after training on DLRSD. We compare against it on iSAID under that cross-dataset protocol, and our method must exceed 39.09% without any training to count as beating it.
The supervised closed-set models give the ceiling, not the target. PDSSNet reports 84.68% mIoU on Vaihingen, 87.55% on Potsdam and 56.10% on LoveDA. These show how much of the open-vocabulary deficit part voting recovers.
Three ablations isolate the key ingredient, and each changes exactly one thing while keeping features, regions and vocabulary fixed:
- Multi-description prompting with max or mean pooling over the same vocabulary but no neutral set and no region voting.
- Region pooling of a single averaged class prompt, which mirrors the mean-prototype design of ViPMoG.
- Part voting without gap centering.
Beating a baseline means a mIoU gain averaged over the eight SegEarth-OV segmentation datasets, larger than the seed-to-seed spread across three runs of the region generator. Separately, a gain in the shadowed stratum must exceed the gain in the lit stratum.
Why not just…?
Why not just prompt with many descriptions per class and take the maximum? This is the most serious threat. It captures much of the "class as many parts" intuition with no new machinery. It lacks two things. There is no neutral sink, so a roof shadow still votes for whichever class has the most shadow-like description, and that is often "road". There is also no region-level aggregation. Ablation (1) tests this directly. If max-pooling matches part voting in the shadowed stratum, our contribution shrinks to the gap-placement finding, and we would report it that way.
Why not just group pixels with better visual features and average the CLIP scores per region? Region averaging collapses tiles, glass and shadow into one mean vector, which is the failure mode we attribute to ViPMoG. Cross the Gap shows that CLIP's image-to-image geometry is poorly calibrated. Even so, strong visual grouping might fix shadows through context alone. Ablation (2), run over the same regions, settles this. If it ties, the text-part vocabulary is not earning its place.
Why not just recalibrate scores with an affine map or a temperature? ModGap's analysis implies that such rescaling preserves rank order and cannot change distance-based grouping. Our "centre at the ranking step only" condition is effectively this alternative. The prediction is that it moves mIoU little, while centering at the assignment step moves it more.
Why not just fine-tune with LoRA or fully on labelled aerial masks? That changes the regime to closed-set and label-dependent. SegEarth-OV reports that its training-free pipeline outperforms fine-tuned open-vocabulary methods on average. The PDSSNet numbers mark what full supervision buys. We include LayerNorm-only tuning as the cheap trained variant, so that any gain from learning is measured separately from the part-voting mechanism.
Decisive Checks
The experiments run on the eight-dataset SegEarth-OV segmentation suite (OpenEarthMap, LoveDA, iSAID, Potsdam, Vaihingen, UAVid, UDD5, VDD), the building and road extraction sets, and LoveDA's urban-to-rural split. The urban-to-rural split tests whether text-defined parts transfer across geographies better than visual statistics do. For reference, CBST reaches 41.32% mIoU there under unsupervised domain adaptation, against an oracle of 46.79%.
Three outcomes would confirm the proposal. Gains must be largest in shadowed and atypical strata. Centering must help at the assignment step and be neutral or harmful at the ranking step, which would reconcile Mind the Gap with ModGap inside one pipeline. Part voting must beat both max-pooled multi-prompting and region-averaged single prompts. The proposal is refuted if the gap-placement interaction is absent, or if ablation (1) matches the full method.
References
- Closing the Modality Gap Aligns Group-Wise Semantics
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- SegEarth-OV: Towards Traning-Free Open-Vocabulary Segmentation for Remote Sensing Images
- TOPIC MODELLING FOR OBJECT-BASED CLASSIFICATION OF VHR SATELLITE IMAGES BASED ON MULTISCALE SEGMENTATIONS
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning
- Foreword to the Special Issue on Information Extraction From High-Spatial-Resolution Optical Remotely Sensed Imagery
- Classification of High Spatial Resolution Remotely Sensed Imagery Based Upon Fusion of Multiscale Features and SVM
- Prototype-Driven Structure Synergy Network for Remote Sensing Images Segmentation
- LOGCAN++: Adaptive Local-Global Class-Aware Network for Semantic Segmentation of Remote Sensing Images
- Embedding Generalized Semantic Knowledge Into Few-Shot Remote Sensing Segmentation
- Bridging the Modality Gap: Training-free Adaptation of Vision-Language Models for Remote Sensing via Visual Prototypes
- Open-Vocabulary Remote Sensing Image Semantic Segmentation
- Mitigating the Modality Gap in Vision–Language Models with Fractal Spectral Geometry
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion