Phonovisual Biases in Language: is the Lexicon Tied to the Visual World?

Andrea Gregor de Varda, Carlo StrapparavaIJCAI 2021 · DOI

TL;DR

This paper investigates the hypothesis that there's a universal, non-arbitrary connection between the sounds of words and their visual referents. Using a deep learning model, they demonstrate that this connection can be learned and generalized across different languages, suggesting a shared symbolic substrate in human language.

What Problem Does This Paper Address?

For decades, the prevailing view in linguistics, rooted in structuralism, has been that the relationship between a word's sound and its meaning is arbitrary. This "arbitrariness of the sign" suggests that words are essentially cultural conventions. However, this perspective struggles to explain how language emerges and interfaces with other cognitive domains. An alternative, "iconicity," proposes that some word-sound relationships are motivated by the intrinsic qualities of sounds themselves. While this idea has historical roots and has been supported by studies on non-words (like the "bouba-kiki" effect), its role in meaningful, cross-linguistic lexicons has been less explored. This paper tackles the question of whether meaningful words, across different languages, exhibit systematic, non-arbitrary connections to the visual attributes of their referents, and if these connections can be learned by a computational model.

Methods

The researchers employed a deep learning approach using a Long Short-Term Memory (LSTM) recurrent neural network. The core idea was to train the network to associate phonetic representations of words with visual representations of their corresponding objects.

Here's a breakdown of the methodology:

  • Phonetic Representation: Words were first transcribed into the International Phonetic Alphabet (IPA) using Epitran. These IPA strings were then converted into sequences of subsegmental articulatory features using PanPhon. This feature-based representation, rather than a simple hot-encoding, allows the model to capture inherent similarities between different sounds (e.g., the shared "voiced" feature between /p/ and /b/). Sequences were padded to a uniform length.
  • Visual Representation: Images were processed through a pre-trained VGG16 Convolutional Neural Network (CNN). The output of the fifth max-pooling layer was used as a proxy for the visual representation of the object, capturing cognitively inspired features.
  • Neural Architecture: An LSTM network was trained to map the phonetic feature sequences to the visual feature vectors. The LSTM's recurrent nature is crucial for processing sequential data like phonetic transcriptions. The model used cosine similarity as its objective function and metric, and the Adam optimizer for training.
  • Experimental Design: To ensure the learned correspondences were not due to etymological relatedness, the study used a strict cross-lingual setup. Concepts and their associated images were divided into training and testing sets. The training set included phonetic representations from five languages, while the test set used a language not present in the training data. This "zero-shot" cross-lingual transfer is key to testing for language-independent generalizations. A randomized baseline was used to compare performance against chance.

Datasets

  • THINGS database: This was the primary dataset, containing 26,107 high-quality naturalistic images depicting 1,854 diverse object concepts. Concepts were sampled from American English.
  • Multilingual Lexicon: The English labels from the THINGS database were translated into five languages from different language families: Arabic (Afroasiatic), Hungarian (Uralic), Indonesian (Austronesian), Vietnamese (Austroasiatic), and Turkish (Turkic). This translation process involved word2word and fastText resources. The final dataset used for experiments comprised 1,161 concrete words across these languages.

Key Insights and Ideas

  • Beyond Arbitrariness: The study provides strong empirical evidence that the lexicon is not purely arbitrary. There are systematic, non-arbitrary relationships between word sounds and the visual properties of their referents.
  • Cross-Linguistic Generalization: The model's ability to perform above chance on a language it was never trained on is a critical finding. This suggests that the learned phonovisual correspondences are language-independent, pointing towards a universal substrate.
  • Deep Learning as a Tool: The research demonstrates the power of deep learning models (specifically LSTMs and CNNs) to uncover complex, cross-modal relationships that are difficult to detect through traditional linguistic analysis alone.
  • Phonetic Features Matter: Representing sounds by their articulatory features, rather than simple phoneme identities, was crucial for the model to learn meaningful similarities between sounds.
  • Iconicity as a Complement: The authors emphasize that iconicity doesn't negate arbitrariness. Instead, they complement each other, enriching the lexicon and potentially aiding language acquisition.

Key Results

The cross-lingual LSTM models consistently outperformed the randomized baselines across all experimental conditions.

  • Statistical Significance: All comparisons between the cross-lingual models and the randomized baselines reached high statistical significance.
  • Effect Size: Measures of effect size (Cohen's d) indicated a medium-to-large difference between the models, with the exception of a marginal difference for the zero-shot transfer to Vietnamese.
  • Cosine Similarity: While specific numbers vary by language fold, the cosine similarity scores for the cross-lingual models were consistently higher than for the randomized models, indicating better alignment between predicted and target visual vectors. For example, for the Arabic fold, the mean cosine similarity was 0.2343 (cl) vs. 0.2243 (r).

Limitations and Future Work

  • Scope of Lexicon: The study focused on a selected subset of concrete nouns. Future work could explore a broader range of vocabulary, including abstract concepts and verbs.
  • Visual Attributes: While the study shows a link between sound and visual features, it doesn't pinpoint which specific visual attributes (shape, size, color, etc.) are most strongly associated with particular phonetic features. Future research could delve into this.
  • Locus of Biases: The paper acknowledges that understanding where these biases originate (innate vs. learned through statistical correlations) is beyond its scope and requires further investigation.
  • Model Complexity: While the LSTM and VGG16 are powerful, exploring other architectures or combinations might yield further insights.
  • Zero-Shot Transfer: While successful, the zero-shot transfer to Vietnamese showed a slightly smaller effect size, suggesting potential nuances in how different language families might encode these biases.

Summary updated

BibTeX
@inproceedings{varda2021phonovisual,
  author = {Andrea Gregor de Varda and Carlo Strapparava},
  title = {{Phonovisual Biases in Language: is the Lexicon Tied to the Visual World?}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/89},
}
Figure