Zero-shot segmentation models label each pixel with a category described only in words, including categories never seen during training. They work well on everyday photos and fail badly elsewhere. Across 22 datasets from medicine, remote sensing, agriculture and engineering, the best such model reaches 38.14 mean intersection-over-union (mIoU, the average overlap between predicted and true masks), against 70.99 for supervised in-domain models. On medical and Earth-monitoring imagery the gap is 30 to 40 points (What a MESS). The usual remedy for a shifted image distribution is adversarial domain adaptation: a second network tries to tell source images from target images, and the feature extractor is trained to fool it. That remedy collides with a finding from open-vocabulary segmentation. Changing the vision-language model's dense features damages its ability to recognize arbitrary words, which is why the strongest method freezes those features and refines only image-text similarity maps (CAT-Seg).
We propose to place the adversarial game somewhere else: on a small vocabulary of mid-level visual attributes (recurring local properties such as "fibrous texture," "bright thin curve," or "blob with dark rim") discovered without labels from source and target images together. Each pixel is described by how strongly it expresses each attribute. Category masks are formed by mixing attribute maps, with weights obtained by projecting the category's text embedding onto the attribute vocabulary. The dictionary is split into "content" attributes, whose spatial maps must look the same in both domains, and "nuisance" attributes, which are allowed to absorb whatever identifies the domain. Only content attributes vote for categories. The backbone stays frozen, so open-vocabulary recognition is not damaged. The alignment is also class-agnostic, so it does not depend on the target domain's category names, which differ from the source's and which language models often handle poorly in specialized domains.
Core Claim
For zero-shot segmentation under severe domain shift with unlabeled target images available, we claim that aligning distributions over a discovered attribute vocabulary beats each of the three obvious alternatives. Aligning dense features damages open-vocabulary recognition. Aligning class-level similarity maps forces the source and target label proportions to match, which causes negative transfer when categories differ. Doing nothing leaves the gap MESS documents. We further claim that the advantage concentrates on shifts that are not purely low-frequency style, such as thermal or other sensor changes, cryo-microscopy, and aerial geometry. For pure color or stain shifts, Fourier-based normalization may already suffice, and we design the experiments to show where that boundary lies.
Relation to Prior Work
Three papers motivate the gap. VGSE discovers attributes by clustering image patches and projecting them into word-embedding space, improving zero-shot classification over word2vec (64.0 vs. 59.6 top-1 on AWA2). Its clusters latch onto environmental shortcuts such as cage wire, it fixes the cluster count by hand, and it never sees a second domain. AttrSeg segments categories through attribute descriptions aggregated by slot attention, reaching 58.7 mIoU on fictional "Fantastic Beasts" categories where name-based models fall below 10. Its attributes come from an LLM or manual curation rather than from the pixels, and it assumes ordinary RGB images. APN shows that localizing attributes with per-attribute prototypes removes spurious correlations (part-localization accuracy rises from 30.7% to 52.8% on CUB), but it needs a fixed human attribute matrix and body-part groupings. We borrow discovery from VGSE, attribute-mediated segmentation from AttrSeg, and local prototypes from APN. We differ from all three in learning the attributes from unlabeled pixels of both domains and in making a subset of them domain-invariant.
Two papers constrain the design. CAT-Seg shows that fine-tuning CLIP's dense features hurts open-vocabulary generalization, which rules out the standard adversarial recipe of pushing a gradient-reversal signal into the backbone. Deep Pre-Alignment makes the parallel point for vision-to-language projectors: training them end to end erodes the language model's existing abilities. Together these push us to keep the encoder frozen and adapt only a small dictionary and projection.
Several papers supply the method. An Adversarial Domain Adaptation Network for Cross-Domain Fine-Grained Recognition shows that global alignment confuses similar subcategories and needs local, part-level structure. That is evidence that alignment should operate on something finer than the whole image, but its closed label set cannot reach new categories. Interpreting CLIP's Image Representation shows that CLIP's representation decomposes into text-describable directions and that removing background-specialized components raises Waterbirds worst-group accuracy from 48% to 73%. This is direct precedent for our content/nuisance split, which learns the split rather than finding it by inspection. The reconstruction-bottleneck attribute learners of Wei et al. supply the reconstruction constraint that keeps the nuisance branch from becoming a trash bin for content.
DIPL is the closest prior pairing of projection and domain invariance. It learns a domain-invariant linear map between visual features and a predefined attribute space, treating seen versus unseen classes as the two domains. It uses no discovered attributes, no spatial maps and no sensor shift, and its gains shrink on fine-grained CUB.
Research Program
We build on a frozen CLIP ViT backbone with CAT-Seg's cost-aggregation head trained on COCO-Stuff, so the starting point is the strongest published model on MESS. Let $f(p)$ be the frozen dense feature at pixel $p$. We learn a dictionary of $K$ unit vectors $u_1,\dots,u_K$ in the image-feature space, initialized by spherical k-means over patch features pooled from source and unlabeled target images. The attribute response map is $a_k(p)=\cos(f(p),u_k)$. The dictionary is split into content atoms $\mathcal{C}$ and nuisance atoms $\mathcal{N}$.
A linear projection $P$ maps content atoms into CLIP's text space. Its job is to bridge the gap that separates image and text clusters even in contrastive models. For each category name with text embedding $t_c$, we solve a small sparse coding problem $w_c=\arg\min_w \lVert t_c - P U_{\mathcal{C}} w\rVert^2+\beta\lVert w\rVert_1$, where $U_{\mathcal{C}}$ stacks the content atoms and $\beta$ sets sparsity. The category map $s_c(p)=\sum_{k\in\mathcal{C}} w_{ck}\,a_k(p)$ enters CAT-Seg's aggregation head in place of, or alongside, the raw cosine cost volume.
Training minimizes
$$\mathcal{L}=\mathcal{L}_{seg}^{src}+\lambda\,\mathcal{L}_{rec}-\mu\,\mathcal{L}_{D}\big(\{a_k\}_{k\in\mathcal{C}}\big),$$
where $\mathcal{L}_{seg}^{src}$ is the source segmentation loss and $\mathcal{L}_{rec}$ requires all $K$ atoms, weighted by their responses, to reconstruct $f(p)$ on both domains. $\mathcal{L}_D$ is the loss of a small convolutional domain classifier that sees only the content-attribute maps; gradient reversal is applied to it. $\lambda$ and $\mu$ weight the reconstruction and adversarial terms. Gradients reach only $U$, $P$ and the aggregation head, never the backbone. Nuisance atoms receive no adversarial pressure and no say in category scores. Reconstruction forces whatever is domain-specific to live somewhere other than the content atoms.
Analysis is part of the program. We name each discovered atom by its nearest text directions, in the manner of the TEXTSPAN decomposition, and visualize its response maps. This tests whether content atoms look like object properties and nuisance atoms look like sensor or background properties. We sweep $K$, because VGSE reports that large cluster counts overfit to seen classes. We also measure how much the target label distribution differs from the source, since any marginal-matching adversary can be hurt by label shift. The attribute level should be less exposed than the class level, but not immune.
Baselines
The naive reference point is frozen CAT-Seg-L applied to each target dataset exactly as in MESS, with no target images used. Its reported MESS mean is 38.14 mIoU (What a MESS); we rerun it on our dataset subset to obtain per-dataset numbers under our evaluation code.
The strongest existing competitor is that same CAT-Seg model given the unlabeled target images our method receives. It is adapted by confidence-thresholded self-training: target pseudo-labels from the frozen model are used to fine-tune the aggregation head and the Query and Value projections that CAT-Seg already permits updating. This is the natural thing a practitioner would do, and it uses the same parameters our method may change. The research file contains no reported result for self-trained CAT-Seg on MESS.
A third baseline is Fourier amplitude normalization of target inputs toward source statistics in the style of FAN-Net, which raised Dice from 0.5047 (nnU-Net) to 0.5591 on ATLAS. We apply it as input preprocessing before frozen CAT-Seg and also combined with our method.
A fourth is an AttrSeg-style variant: our pipeline with the discovered dictionary replaced by LLM-generated attribute text embeddings for each target category, aligned the same way.
All methods get the same backbone, source data, unlabeled target pool, number of training steps, and a tuning budget of the same number of configurations. Hyperparameters are selected on a small labeled target validation split that is identical across methods and reported as such, since true unsupervised model selection is itself an open problem. Beating the strongest competitor means a higher mIoU than self-trained CAT-Seg averaged over three seeds on the sensor-shift and microscopy datasets, with no loss larger than one point on COCO-Stuff and ADE20K-847. Those two datasets check that open-vocabulary ability survived.
The ablations that isolate the key ingredient move the adversary while holding everything else fixed. In the first, the discriminator reads the dense backbone features through a trainable adapter. In the second, it reads the class-level cost volume. In the third, the content/nuisance split is removed, so all atoms are aligned. A fourth ablation removes the adversary entirely, leaving the discovered attributes alone.
Why not just…?
Why not just run the foundation model zero-shot? MESS already answers this: the best zero-shot model trails supervised models by roughly 33 points on average and by 30 to 40 on medical and Earth imagery. Grounded-SAM does worse still, so larger mask proposers do not close the gap (What a MESS). It remains our naive reference.
Why not just normalize image style in the Fourier domain? This is a real threat for shifts that are mostly color, stain or illumination. Fourier amplitude carries style and phase carries structure, and FAN-Net shows that normalizing amplitude helps across clinical sites. We do not argue around it. We group target datasets by shift type and predict that Fourier normalization matches our method on low-level shifts like Kvasir-Inst and loses on thermal PST900, CryoNuSeg and aerial ISPRS Potsdam, where the shift changes which local structures appear, not only their color. If Fourier normalization matches us everywhere, the attribute-level claim is false.
Why not just self-train on pseudo-labels? Self-training reinforces whatever the frozen model already gets right, and at 30-40 points behind supervision in these domains, its confident pseudo-labels are few and biased toward easy classes. It is our strongest baseline precisely because it is simple. The decisive comparison is on datasets where the zero-shot starting point is lowest.
Why not just prompt an LLM for attribute descriptions? Evolving Interpretable Visual Classifiers shows that LLM-written attributes fail where web text is sparse: visually evolved descriptors beat text-prompted ones by 18.4% on specialized iNaturalist families. VGSE likewise shows that visually grounded embeddings beat word embeddings. Our LLM-attribute variant measures this directly within the identical alignment pipeline.
Why not just align the backbone features adversarially? CAT-Seg reports that fine-tuning CLIP's dense features harms open-vocabulary generalization. The dense-feature ablation, scored on both target mIoU and COCO-Stuff/ADE20K retention, tests whether that harm appears under adversarial training too.
Proposed Experiments and Decisive Checks
The main experiment trains on COCO-Stuff with labels. Unlabeled training images are drawn from the MESS datasets with the largest zero-shot gaps: Kvasir-Inst, CHASE DB1 and CryoNuSeg for medicine; WorldFloods and ISPRS Potsdam for Earth monitoring; PST900 for thermal; and DeepCrack for engineering. We report per-dataset and mean mIoU on held-out target test splits, grouped by shift type. We report retention on COCO-Stuff and ADE20K-847 to confirm that open-vocabulary recognition is intact.
The first decisive check is the position-of-the-adversary ablation. If aligning the class-level cost volume does as well as aligning attributes, the attribute layer is unnecessary. We expect class-level alignment to degrade most on target datasets whose label proportions differ most from the source, and we test this by subsampling target images to vary those proportions on purpose.
The second check is the content/nuisance split. We train a probe to predict the domain from content-atom maps and from nuisance-atom maps. We expect near-chance accuracy on content maps and high accuracy on nuisance maps. We then ablate the nuisance atoms at test time, by analogy with the Waterbirds head-ablation result.
The third check is interpretability as a sanity test. We name atoms via nearest text directions and check on CUB-200, which appears in MESS and has part annotations, whether content atoms localize parts, using the PCP metric APN reports (52.8%). This does not compete with APN, which uses human attributes; it verifies that the discovered vocabulary is about objects rather than backgrounds.
The claim stands if three things hold together: our method beats self-trained CAT-Seg on the sensor and microscopy datasets, the attribute-level adversary beats both feature-level and class-level adversaries, and retention on COCO-Stuff and ADE20K holds.
References
- What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation
- CAT-Seg: Cost Aggregation for Open-vocabulary Semantic Segmentation
- VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning
- AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation
- Attribute Prototype Network for Zero-Shot Learning
- Deep Pre-Alignment for VLMs
- An Adversarial Domain Adaptation Network For Cross-Domain Fine-Grained Recognition
- Interpreting CLIP's Image Representation via Text-Based Decomposition
- A$^2$-Net: Learning Attribute-Aware Hash Codes for Large-Scale Fine-Grained Image Retrieval
- Domain-Invariant Projection Learning for Zero-Shot Recognition
- Fan-Net: Fourier-Based Adaptive Normalization for Cross-Domain Stroke Lesion Segmentation
- Evolving Interpretable Visual Classifiers with Large Language Models