Where Does Sound Symbolism Live in a Multimodal Embedding Space?

Vision-language and audio-visual models reproduce a classic finding about language: people match made-up words like "kiki" to spiky shapes and "bouba" to rounded ones. CLIP, a model trained to place images and their captions near each other, shows the effect for written pseudowords. Models trained on spoken image captions show it more strongly for audio. Researchers have read these results as evidence that the models learned a link between word form and perceptual meaning across modalities. These same embedding spaces have a well-documented quirk, called the modality gap: images, text, and audio sit in separate clusters, offset from one another by a large, nearly constant vector. No study has asked where the sound-symbolic signal sits relative to that geometry. It could lie in the directions that images and words actually share. It could also lie in directions private to one modality, such as a text-side "spelling" subspace that reads letter strings.

This note proposes to locate the signal geometrically and then move it on purpose. We split each model's space into a part shared across modalities and a part private to each modality, using the affine-subspace picture of the modality gap. We then measure how much of the sharp–round direction, and a few other sound-symbolic dimensions, falls in each part. Finally, we test whether training objectives that close the gap shift the signal from private to shared directions. If CLIP's kiki/bouba effect turns out to sit mostly in text-private, spelling-like directions, then much of the "cross-modal iconicity" reported for text-image models is orthographic reading, not grounding. If gap-closing training moves the signal into shared directions, the modality gap is a measurable obstacle to cross-modal form–meaning alignment.

Core Claim

Existing sound-symbolism probes cannot detect the modality gap, because of how they are built. Kiki or Bouba? defines a sharp–round axis $\gamma = \mu_{\text{sharp}} - \mu_{\text{round}}$, where $\mu$ denotes the mean embedding of sharp or round adjectives. It scores each item $x$ by its projection $s(x) = \langle x, \gamma \rangle$. Suppose a modality's embeddings are shifted by a constant vector $c$, as post-hoc gap removal does. Then $\gamma$ is unchanged and

$$s(x - c) = s(x) - \langle c, \gamma \rangle,$$

which is the same constant for every item in that modality. Every within-modality ROC-AUC is therefore unchanged. The Geometric and Phonetic Scores of Measuring Sound Symbolism in Audio-visual Models project items of one modality onto an axis from another, and they are also invariant. The gap can affect these probes through only two routes. The first is renormalization after centering, which rescales each item by its own factor $1/\lVert x - c \rVert$. The second is training, which changes which directions carry meaning in the first place.

Our claim is that the question worth asking is not "does closing the gap change the AUC?" It is this: is the sound-symbolic direction carried by the subspace that modalities share, or by modality-private dimensions? Text-image models and speech-image models are hypothesized to differ sharply on this point, and gap-closing training should change it.

Prior Work and How This Differs

Three papers motivate the work by documenting the effect without examining geometry. Kiki or Bouba? probes CLIP and Stable Diffusion with 648 pseudowords. Measuring Sound Symbolism in Audio-visual Models probes eight audio-visual models with synthesized audio pseudowords and rendered shapes. It finds that spoken-caption-grounded models such as SpeechCLIP succeed while self-supervised models like AV-HuBERT and CAV-MAE sit at chance. Do Language Models Associate Sound with Meaning? introduces LEX-ICON, which covers 25 semantic scales in four languages. It probes generative multimodal models by prompting and attention analysis, not through shared embedding geometry. None of the three measures, removes, or decomposes the modality gap. We reuse their stimuli and probes and add the geometric decomposition.

Two papers supply our method. Is the Modality Gap a Bug or a Feature? shows that contrastive training leaves each modality in its own affine subspace, with the gap vector orthogonal to both. That picture gives us the shared/private split. Disentangling visual and written concepts in CLIP shows that CLIP's spelling ability is concentrated in about 64 of 512 dimensions. That subspace is our concrete candidate for an orthographic carrier of kiki/bouba.

Three papers supply the interventions and also constrain the design. Closing the Modality Gap for Mixed Modality Search removes the gap post hoc by mean-centering (GR-CLIP). AlignCLIP uses shared parameters and a semantic separation loss. Closing the Modality Gap Aligns Group-Wise Semantics (ModGap) uses direct pair alignment plus repulsion between concept centroids. ModGap also proves that rank-based retrieval is largely insensitive to the gap while distance-based uses are not. Our invariance argument applies that reasoning to sound-symbolism probes. It rules out the naive design of "center and re-probe" and redirects effort toward retraining.

Modelling Form-Meaning Systematicity with Linguistic and Visual Features is the closest earlier attempt to put form–meaning systematicity in multimodal space. It used separately trained text and CNN features fused by distance averaging, and found that visual information exposed phonaesthemes missed by text alone. It predates joint contrastive spaces and so says nothing about the gap.

The linguistics literature constrains how much signal to expect. Meaning to Form finds that lexical systematicity reduces uncertainty about word form by under one percent. Whole-lexicon analyses would therefore be underpowered, and we restrict the main analysis to controlled pseudowords and mimetic words, where the effect is large.

Research Program

The first step is the decomposition. For each model we embed a calibration set of paired data in every modality and center each modality by its own mean. We fit a principal subspace per modality, keeping enough components for 95% of variance. We then compute principal angles between the text and image subspaces, and between the audio and image subspaces, using the singular value decomposition of the product of their bases. Directions whose principal angle falls below a threshold form the shared subspace, and the remainder in each modality form its private subspace. The calibration set is COCO captions for CLIP and spoken COCO captions where available for SpeechCLIP.

We report results across a sweep of thresholds rather than at a single cutoff. For each sound-symbolic axis we measure the fraction of its norm in the shared and private parts. We also measure each part's principal-angle overlap with the 64-dimensional spelling subspace. We reproduce that subspace with the projection of Disentangling visual and written concepts in CLIP, re-learning it with their procedure if the projection is not available.

The second step is causal ablation. We project out the shared part, the private part, the spelling subspace, or a random subspace of matched dimension. After each ablation we re-run four probes: within-text pseudoword AUC, cross-modal AUC (images scored on a text-derived axis), audio-on-visual Geometric Score, and image-on-audio Phonetic Score. The stimuli are the 648 pseudowords and Stable Diffusion renders from Kiki or Bouba? and the Texas AV set of 500 images and 3,888 audio pseudowords from Measuring Sound Symbolism in Audio-visual Models. The models are CLIP (text and image), SpeechCLIP and FaST-VGS (speech and image), and ImageBind (all three modalities, which lets us compare text and audio routes to the same image space).

Our prediction for CLIP is that within-text AUC survives removal of the shared part and collapses when the spelling subspace is removed, while cross-modal AUC tracks the shared part. For SpeechCLIP we predict the signal is mostly shared.

The third step is the training intervention. We fine-tune CLIP ViT-B/32 on CC3M under three conditions with identical data, steps, and hyperparameter search budget. The first uses the plain contrastive loss. The second adds ModGap's pair-alignment and centroid-uniformity losses. The third uses AlignCLIP's shared projection with its intra-modality separation loss. We then repeat the decomposition and probes. The plain fine-tune isolates the effect of gap-closing itself from the effect of additional training.

The fourth step extends beyond shape. From LEX-ICON we take the scales with a visual counterpart we can render, namely big–small, bright–dark, and sharp–round. We build adjective-derived axes for each and repeat the shared/private analysis on the natural mimetic words and pseudowords. We use text and IPA inputs for CLIP-family models and audio for ImageBind. This tests whether the localization pattern generalizes or is specific to kiki/bouba.

Baselines

The naive reference point is a letter-identity classifier trained on half the pseudowords. Because sharp and round pseudowords are built from disjoint letter sets, such a classifier should be nearly perfect. That shows classifiability alone proves nothing. Every probe here must therefore be zero-shot, with its axis derived from adjectives or images rather than from the pseudowords.

The strongest existing text-image result is the unmodified projection probe of Kiki or Bouba?, which reports ROC-AUC 0.77 for CLIP and 0.74 for Stable Diffusion. We re-run it on our exact checkpoints so that every ablation is compared against our own reproduced number, not the published one. For audio, the strongest result is from Measuring Sound Symbolism in Audio-visual Models, which reports AUC up to 0.87 for spoken-caption-grounded models. On LEX-ICON the human-alignment reference is Qwen2.5-Omni-7B, with Pearson 0.579 against human judgments in Do Language Models Associate Sound with Meaning?. That number comes from a generative prompting setup, so we cite it as context rather than as a head-to-head competitor.

The post-hoc gap baseline is GR-CLIP centering from Closing the Modality Gap for Mixed Modality Search, applied with and without renormalization. Without renormalization it should reproduce the raw AUC exactly, and any deviation indicates a bug. With renormalization, any change measures the only post-hoc channel through which the gap can matter.

The key ablation is the plain-contrastive fine-tune against the ModGap and AlignCLIP fine-tunes. For context on how large these interventions are: ModGap reports reducing the gap from 0.47 to 0.03 with stable R@1. AlignCLIP reports raising average paired cosine similarity from 0.47 to 0.67. Beating the baseline here means the gap-closing fine-tune raises cross-modal AUC on sharp–round by more than the plain fine-tune. It also means the shared-norm fraction rises, with both effects exceeding seed-to-seed variance across three seeds. The random-subspace ablation of matched dimension is the control for every projection-removal claim.

Why not just…?

Why not just mean-center the embeddings and re-run the kiki/bouba probe? The display equation above shows that this cannot change any difference-based AUC unless the embeddings are renormalized. A null result would therefore be guaranteed and uninformative. We still run it, as a sanity check and to measure the small renormalization effect. Its value is in showing that earlier probes were blind to the gap, not in answering whether the gap matters.

Why not just apply the "Forget to Spell" projection and check whether CLIP's effect disappears? This is a real threat to the decomposition's added value. If removing the 64-dimensional spelling subspace from Disentangling visual and written concepts in CLIP kills within-text AUC, then orthography explains the text-side effect without any shared/private machinery. The spelling ablation does not, however, say whether the remaining cross-modal signal (images scored on text axes) lives in shared directions. It also cannot be applied to SpeechCLIP, which has no text input to spell. We run it as one arm of the ablation, and we will report plainly if it accounts for most of the effect.

Why not just compare AUCs across architectures, as earlier work did? Comparing CLIP with SpeechCLIP confounds input form, training data, and gap geometry at the same time. Measuring Sound Symbolism in Audio-visual Models shows that grounded models outperform self-supervised ones, but it cannot say why. The matched fine-tunes hold data and architecture fixed and vary only the gap-closing objective, which is what attributing a cause requires.

Decisive Checks

The study rests on three outcomes. First, under the centering-only baseline, within-modality AUCs must be invariant up to numerical precision, which validates the measurement setup. Second, the localization hypothesis is supported if CLIP's sharp–round axis has a markedly larger private, spelling-overlapping fraction than SpeechCLIP's audio axis. In that case, removing the private part should cut CLIP's within-text AUC far more than its cross-modal AUC, while random-subspace removal of matched size changes neither. The hypothesis fails if the shared fractions are indistinguishable across model families. Third, the gap-as-obstacle hypothesis is supported only if ModGap or AlignCLIP fine-tuning raises both the shared fraction and cross-modal AUC above the plain fine-tune. Such a result would add a perceptual-semantic finding to the ModGap camp's clustering gains. It would also check whether the orthogonal gap valued for robustness in Is the Modality Gap a Bug or a Feature? costs anything for cross-modal form–meaning transfer. If the LEX-ICON size and brightness axes reproduce the same pattern, the conclusion extends beyond the single kiki/bouba contrast that all prior multimodal work has studied.