Evolving How Many Visual Anchors Each Abstract Concept Needs

Image-text models such as CLIP recognize "apple" well and "justice" poorly. A common explanation is that abstract words have no single thing to point at. This proposal tests a more specific version of that explanation. The trouble may come less from the concept itself and more from the shape of its representation. Every concept, concrete or abstract, is usually stored as one point in embedding space and compared to an image by one similarity score. That is effectively a hard clustering decision: each concept gets exactly one prototype.

We propose to give each concept its own small grounding structure, and to let a genetic algorithm search for it per concept. The structure has three parts. The first is a set of concrete "anchor" phrases, such as "open sky" or "broken chains" for freedom. The second is a soft weighting over those anchors. The third is a temperature that controls whether the anchors are pooled sharply, which behaves like picking the single closest prototype, or diffusely, which behaves like averaging a whole neighborhood. The search also carries its own mutation rates, which adapt during the run.

The prediction is concrete and falsifiable. Concrete concepts should settle on one anchor with sharp pooling. Abstract concepts should settle on several anchors with soft pooling. The amount of structure each concept ends up with should track human abstractness ratings that the search never sees.

Core Claim

The claim is that grounding structure should be a property of each concept, not a global hyperparameter. Searching for it per concept, with a rank-based, human-aligned fitness, should improve how well a frozen model's judgments of abstract concepts in images match people's. It should do so without hurting concrete recognition.

The motivating evidence comes from distributional semantics. Semantic Similarity Computation for Abstract and Concrete Nouns Using Network-based Distributional Semantic Models finds that concrete and abstract nouns need different similarity metrics. Concrete nouns are best served by a "maximum sense" metric that peaks at 30–40 neighbors. Abstract nouns are best served by an attributional-neighborhood metric that peaks near 10. If that topological difference carries over into vision-language embeddings, then no single global choice of prototype count or pooling sharpness can be right for both kinds of concept.

Formally, for an image with frozen image embedding $f(x)$, concept $c$ is scored as

$$ s_c(x) = \tau_c \log \sum_{j \in A_c} w_{c,j}\, \exp\!\big(\cos(f(x), a_j)/\tau_c\big). $$

Here $A_c$ is the concept's chosen subset of anchors, drawn from a shared bank of text embeddings $a_j$ of high-concreteness phrases. The $w_{c,j}$ are nonnegative weights summing to one, and $\tau_c$ is the concept's pooling temperature. As $\tau_c \to 0$ the score becomes the similarity to the single best anchor, which is hard assignment. As $\tau_c$ grows, the score approaches a weighted average over the anchor neighborhood. One genome encodes $A_c$, $w_c$, $\tau_c$, and a per-concept mutation rate $\mu_c$.

Relation to Prior Work

Two lines of work already move away from single prototypes, but both fix how much structure a concept gets.

Decoupling Primitive with Experts (EVA) routes tokens to a mixture of low-rank adapters, so that a state like "old" can take different visual forms. However, the number of experts is fixed in advance and trained by backpropagation.

Parts of Speech–Grounded Subspaces in Vision-Language Models separates content from style in CLIP space by projecting onto learned subspaces. Its authors flag the hand-chosen subspace dimension $k$ as an open limitation.

Both papers motivate our proposal. We differ in making the amount of structure, meaning the number of anchors and the pooling sharpness, a per-concept quantity that is searched for rather than set.

A second line turns abstract language into concrete pictures. ViPE distills an LLM into a model that rewrites metaphors as concrete visual prompts. Generative Imagination Elevates Machine Translation imagines visual features to help translation, but reports a persistent gap on political and economic concepts. We reuse their intuition that abstract concepts can be reached through concrete pieces. We replace one generated image with a weighted set of anchors, because the network-semantics result above suggests abstract meaning is spread across a neighborhood rather than sitting in any single literal proxy.

Evolutionary search has been applied to vision-language models before, but on prompts and code. Evolutionary Prompt Optimization Discovers Emergent Multimodal Reasoning Strategies in Vision-Language Models evolves system prompts and explicitly reports no gains on abstract, theoretical subtasks. EvoVLMA evolves adaptation code for CLIP on concrete few-shot benchmarks. These papers constrain the design: text-level search alone has already failed on abstraction. We therefore search the representation's structure directly, not the text that steers it.

The adaptive-control side supplies the method.

  • Self-adaptation of Mutation Rates in Non-elitist Populations proves exponential speedups when mutation rates are stored in the genome and evolve under selection, on landscapes whose phases need different rates. Its proofs cover only toy pseudo-Boolean functions.
  • Our setting is a natural empirical test outside those toy functions. Concepts differ in the search difficulty of their landscapes, so a rate that is right for "apple" should be wrong for "melancholy."
  • AdaEvolve offers a competing form of control: a running average of recent improvement steers exploration, and a bandit allocates compute across subpopulations. We use it as an alternative controller, not as our core mechanism.

Research Program

Anchor bank. We build a bank of a few thousand concrete phrases, selected for high concreteness and encoded with a frozen CLIP text encoder. Restricting anchors to concrete phrases is deliberate: it forces every abstract concept to be grounded through visually meaningful pieces. It also makes the evolved anchor sets readable, so a human can inspect what "freedom" was grounded in.

Fitness. The fitness for concept $c$ combines two terms computed on the training split of AbsVis:

  • the rank of human-attributed images under $s_c$, compared against other concepts;
  • agreement with the human preference judgments, scoring the concept-image fit of each judged pair.

Both terms are rank-based and non-differentiable. Fitness is evaluated on cached embeddings, so each evaluation costs a few matrix products, not a forward pass through the model.

Search operators. Each concept runs its own population. Mutation swaps, adds, or removes anchors with probability $\mu_c$, perturbs weights and temperature, and also mutates $\mu_c$ itself, following the self-adaptive scheme. Crossover mixes anchor subsets between parents.

Overfitting control. The anchor count carries a small per-anchor penalty, so that a concept only gains anchors when they improve held-out fitness. AbsVis is small (675 images), so we use image-disjoint splits and nested validation. As optional extra positive examples, we use tag-matched YFCC100M images, the same source AbsVis drew from, and report results with and without them.

Analysis. For each concept we record the evolved anchor count, temperature, and final mutation rate. We then correlate each with Brysbaert abstractness ratings, which the search never sees.

Baselines

Zero-shot CLIP. The naive reference point is frozen CLIP scoring each concept with a single prompt, "a photo that expresses [concept]." This is the one-prototype, hard-assignment default the proposal argues against.

AbsVis DPO-tuned VLMs. The strongest existing system on the target benchmark is the preference-tuned VLMs from AbsVis. The authors report "consistent improvements" in human alignment, but I found no reported result on Lacuna for a specific number. We will therefore retrain or re-evaluate their DPO setup on our identical image-disjoint split and score it with our metrics. It uses far more compute than our method, so we report its compute alongside its accuracy. Beating it means matching or exceeding its preference agreement while training no model weights.

Soft k-means / Gaussian mixture. For each concept, we fit soft k-means or a Gaussian mixture to CLIP embeddings of its positive training images, choosing $k$ per concept by cross-validation over a range of 1 to 8.

Sparse coding with a differentiable loss. We fit lasso-style sparse weights over the same anchor bank, trained with a smoothed pairwise preference loss by gradient descent.

These two gradient- and EM-based baselines are the most dangerous competitors. Both already make structure concept-specific without any genetic algorithm. They get the same anchor bank, the same splits, and the same number of hyperparameter trials as the genetic algorithm gets fitness evaluations.

ViPE rewriting. We rewrite each concept into concrete prompts with ViPE, score images by CLIP similarity to the rewritten prompts, and average over several samples.

Ablations. Four ablations each isolate one ingredient:

  • the best single global anchor count and temperature, tuned on the training split for all concepts at once, which isolates per-concept structure;
  • temperature fixed near zero, which isolates soft versus hard pooling;
  • a fixed, grid-tuned mutation rate, which isolates self-adaptation;
  • an AdaEvolve-style improvement-signal controller in place of genome-encoded rates.

Beating an ablation means a gain on held-out preference agreement whose bootstrap confidence interval over concepts excludes zero.

Why not just…?

Why not just fit a per-concept mixture model and pick $k$ by cross-validation? This is the real threat. If it matches the genetic algorithm, the honest conclusion is that per-concept structure matters and the search method does not. The proposal is set up so that this outcome is still informative: the abstractness-correlation analysis is run on whichever method's per-concept structure fits best. Our reason to expect a gap is that mixtures fit on image embeddings model where positive images lie, not which concepts humans prefer for an image. AbsVis's finding that overlap between human and VLM attributions drives preference suggests those two objectives can diverge. The matched-budget comparison settles this directly.

Why not just train sparse anchor weights by gradient descent? The anchor subset is discrete, and the count penalty and rank-based fitness need to be smoothed before gradients can flow through them. EvoVLMA's 1.91-point gain over the hand-designed APE adapter in 8-shot settings (EvoVLMA) suggests that searched structure can beat engineered differentiable designs. That evidence comes from a different task, so the lasso baseline is what actually decides the question here.

Why not just rewrite abstract words into concrete prompts with an LLM? This collapses each concept to whatever a single rewrite produces. ImagiT reports that political and economic concepts stay hard to visualize even with generative imagination, and ViPE's own human-preference rate (33.6%, versus 38.7% for human experts) shows rewrites remain imperfect. Our ViPE baseline measures this alternative directly. We also break results down by abstractness tertile, to check whether rewriting fails specifically on the most abstract concepts.

Why not just use one well-tuned global setting? That is exactly what the first ablation tests. If a single global anchor count and temperature performs as well as per-concept structure, the core claim is false.

Proposed Experiments

Benchmark and metrics. The primary benchmark is AbsVis: 675 images, 10,125 human concept-explanation pairs, and 2,680 preference judgments (AbsVis). We use five image-disjoint folds. The metrics are:

  • recall at 1, 5, and 10 of human-attributed concepts per image;
  • preference-pair agreement;
  • results broken down by abstractness tertile.

Concrete sanity check. Following Parts of Speech–Grounded Subspaces in Vision-Language Models, which reports ImageNet zero-shot accuracy rising from 56.5% to 57.8% with base CLIP, we run the method on ImageNet class names. Two outcomes are required. Accuracy must not fall below the zero-shot baseline. Most classes should evolve a single anchor with low temperature. If concrete classes instead grow many diffuse anchors, the method is fitting noise, not concept structure.

Semi-abstract check. We score state words (adjectives) from MIT-States with frozen CLIP, as a middle ground between concrete objects and abstract nouns. EVA's 23.2% open-world unseen accuracy is cited as context only. That figure comes from a fully trained compositional system, so it is not a fair target for a training-free scorer.

Decisive checks. The proposal succeeds if three things hold:

  • per-concept evolved structure beats the best global setting on held-out preference agreement;
  • the evolved anchor count and temperature correlate positively with Brysbaert abstractness, with Spearman correlation significant across concepts, even though that signal was never used;
  • self-adapted mutation reaches equal fitness with fewer evaluations than the fixed-rate run, with the largest savings on abstract concepts.

The last check is the empirical counterpart of the theory's multi-phase prediction.

Failure interpretation. If the first check fails, the concreteness gap is not about representational shape. If only the comparison against the mixture and lasso baselines fails, the contribution narrows to a measurement result: per-concept structure helps and tracks abstractness. We would report it as that, not as evidence that the genetic algorithm is needed.