What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language models

Tessa Verhoef, Kiana Shahrasbi, Tom Kouwenhoven

Abstract

Humans have clear cross-modal preferences when matching certain novel words to visual shapes. Evidence suggests that these preferences play a prominent role in our linguistic processing, language learning, and the origins of signal-meaning mappings. With the rise of multimodal models in AI, such as vision- and-language (VLM) models, it becomes increasingly important to uncover the kinds of visio-linguistic associations these models encode and whether they align with human representations. Informed by experiments with humans, we probe and compare four VLMs for a well-known human cross-modal preference, the bouba-kiki effect. We do not find conclusive evidence for this effect but suggest that results may depend on features of the models, such as architecture design, model size, and training details. Our findings inform discussions on the origins of the bouba-kiki effect in human cognition and future developments of VLMs that align well with human cross-modal associations.

Introduction

In cognitive science, the bouba-kiki effect is a classic example of cross-modal association. When shown a jagged shape and a rounded shape and asked which is "Kiki" and which is "Bouba," roughly 95% of humans across cultures pick the jagged one for Kiki. This suggests that human language isn't entirely arbitrary; some sounds just "feel" like certain shapes.

As we move toward more advanced Vision-and-Language Models (VLMs), a critical question arises: Do these models encode the same visio-linguistic biases as humans? If they do, it might mean they are learning deeper patterns of human cognition. If they don't, it highlights a fundamental gap in how AI "perceives" the relationship between sight and sound—especially since these models are trained on text and images but usually lack actual auditory experience.

Methodology and Stimuli

The researchers probed four prominent VLMs: CLIP, ViLT, BLIP2, and GPT-4o. To test them, they used a "zero-shot" classification approach, extracting the model's internal probability scores for specific labels when presented with an image.

Examples of curved and jagged visual stimuli used to probe cross-modal associations.

The study utilized several sets of visual stimuli, including original hand-drawn shapes from historical studies and newly generated random shapes (both curved and jagged). The labels used were a controlled set of 36 syllables categorized by phonetic features: Sonorant-Rounded (e.g., loo, nah—expected to match curved shapes) and Plosive-Non-Rounded (e.g., kee, puh—expected to match jagged shapes). The team employed Bayesian multilevel linear models to determine if the shape of the image significantly predicted the probability the model assigned to a specific type of syllable.

Key Results

The findings suggest that while some models show flashes of human-like association, the effect is inconsistent and highly sensitive to prompting.

  • CLIP and GPT-4o: These models were significantly more likely to pair jagged images with plosive consonants (like 'k' or 't') and curved images with sonorant consonants (like 'm' or 'l').
  • ViLT and BLIP2: These models showed no consistent evidence of the effect. In some cases, ViLT even showed a "tiny" interaction in the opposite direction of human preference.
Comparison of model performance in matching curved and jagged images to phonetic features across different VLMs.

Interestingly, when testing longer words (like nohmoh vs. tepete), GPT-4o’s preference for the "correct" human mapping disappeared. It tended to prefer certain syllables regardless of the shape shown, indicating that the bias is a "weak signal" in the models' latent space.

Discussion: Architecture and Perception

Several factors contribute to the inconsistent results observed across models:

  1. Architecture Matters: The "modality gap"—how a model bridges vision and text—is crucial. CLIP, which uses a dual-stream architecture with modality-specific attention, showed the strongest alignment. BLIP2, despite its size, failed, likely because its "Q-Former" bridge doesn't promote these specific associations.
  2. Texture vs. Shape: Many VLMs (especially those based on CNNs or standard Vision Transformers) are biased toward texture rather than global shape. Since the bouba-kiki effect is about the contour of an object, models focusing on local pixel patterns might miss the association entirely.
  3. Tokenization and Lack of Audio: Language models process text in "tokens," not phonemes. Since these models never "hear" the words, they rely entirely on orthography, which is an indirect way to measure a phonetic effect. Additionally, the "domain shift" from realistic training photos to synthetic line drawings might hinder performance.

Conclusion

The bouba-kiki effect in AI is not robust. While CLIP and GPT-4o showed the effect when forced to choose, the signal often vanished when looking at full probability distributions. This suggests that AI "understanding" of sound-symbolism remains superficial. Future work should investigate models on more naturalistic, 3D-rendered objects and explore how tokenization influences the perception of sound-symbolism to better align AI with human cross-modal associations.

Summary updated

Abstract

Humans have clear cross-modal preferences when matching certain novel words to visual shapes. Evidence suggests that these preferences play a prominent role in our linguistic processing, language learning, and the origins of signal-meaning mappings. With the rise of multimodal models in AI, such as vision- and-language (VLM) models, it becomes increasingly important to uncover the kinds of visio-linguistic associations these models encode and whether they align with human representations. Informed by experiments with humans, we probe and compare four VLMs for a well-known human cross-modal preference, the bouba-kiki effect. We do not find conclusive evidence for this effect but suggest that results may depend on features of the models, such as architecture design, model size, and training details. Our findings inform discussions on the origins of the bouba-kiki effect in human cognition and future developments of VLMs that align well with human cross-modal associations.
BibTeX
@inproceedings{verhoef2024what,
  author = {Tessa Verhoef and Kiana Shahrasbi and Tom Kouwenhoven},
  title = {{What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure