Abstract
Character embeddings in Natural Language Processing (NLP) models are often treated as "black boxes." This study investigates whether these learned representations capture human-like perceptual features such as shape, sound, and color across five languages (English, Dutch, Spanish, Korean, and Japanese). By comparing learned embeddings from PPMI, LSTM, and Transformer models to perceptual representations using Representational Similarity Analysis (RSA) and probing classifiers, we demonstrate that models "rediscover" phonological and visual properties inherent in different writing systems. Notably, sound is a primary signal in alphabetic scripts, while the featural nature of the Korean Hangul script leads to high correlations with character shape.
Introduction
While character-level language models (LMs) are widely used for their robustness and ability to handle complex morphology, our understanding of what these models actually represent at the character level remains limited. Most interpretability research focuses on word or sentence levels, often within an "anglocentric" framework. This paper addresses this gap by investigating whether learned character representations align with human perceptual dimensions—specifically visual shape, phonological sound, and synesthetic color associations—across diverse scripts including Latin, Hangul, and Hiragana.
Methodology
The study compares three types of learned textual embeddings against three types of perceptual representations:
Learned Textual Embeddings
- PPMI: A count-based model using bigram contexts.
- LSTM: A recurrent neural network trained for next-character prediction.
- Transformer: A self-attention-based model trained for next-character prediction.
Perceptual Representations
- Sound: Characters were mapped to phonological features (e.g., voicing, place of articulation) using the International Phonetic Alphabet (IPA) and the WikiPron dictionary.
- Shape: Characters were rendered as grayscale images and flattened into vectors to capture visual similarity.
- Color: Data from synesthetes (individuals who associate letters with specific colors) were used to create color-based vectors.
The researchers employed Representational Similarity Analysis (RSA) to correlate the distance matrices of learned vs. perceptual embeddings. Additionally, probing classifiers (logistic regression) were used to determine if embeddings contained sufficient information to predict specific phonological traits like vowel/consonant distinction.
Key Insights and Results
The analysis revealed several critical findings regarding how models perceive different scripts.
Sound as a Primary Signal
For alphabetic scripts (English, Dutch, Spanish), sound is the strongest correlate. Models essentially learn the phonological structure of a language simply by observing character distributions. RSA correlations with sound were particularly high in Spanish (reaching 0.60 for PPMI), likely due to its "shallow" orthography where letters consistently map to sounds.
The Featural Nature of Korean
A standout result was the high correlation between the Korean (Hangul) Transformer model and character shape (0.51), compared to nearly zero (0.03) for English. This aligns with the linguistic design of Hangul, where consonant shapes mimic the physical position of the vocal apparatus.
Probing and Generalization
Probing classifiers achieved near-perfect F1 scores (0.97–1.00) for distinguishing vowels from consonants across all models. Furthermore, a classifier trained on Spanish "voicing" could identify vowels with an F1 of 0.89, suggesting the models learn a broader concept of "sonority."
Discussion and Future Work
The results suggest that character embeddings are not arbitrary but reflect the underlying perceptual and linguistic properties of the scripts they represent. While color associations showed moderate correlations—likely reflecting the "distinctiveness" of certain characters—the strongest signals came from sound and, in specific cases, shape.
Limitations include the relatively small size of synesthesia datasets and the use of simplified shape representations. Future work could involve using Convolutional Neural Networks (CNNs) for more sophisticated visual patterns or incorporating context-dependent pronunciation. These perceptual representations could eventually serve as pre-trained features to improve model robustness in low-resource settings or for tasks involving orthographic errors.
Summary updated