Abstract
In this paper, we introduce a new task called synesthesia detection, which aims to extract the sensory word of a sentence, and to predict the original and synesthetic sensory modalities of the corresponding sensory word. Synesthesia refers to the description of perceptions in one sensory modality through concepts from other modalities. It involves not only a linguistic phenomenon, but also a cognitive phenomenon structuring human thought and action, which makes it become a bridge between figurative linguistic phenomenon and abstract cognition, and thus be helpful to understand the deep semantics. To address this, we construct a large-scale human-annotated Chinese synesthesia dataset, which contains 7,217 annotated sentences accompanied by 187 sensory words. Based on this dataset, we propose a family of strong and representative baseline models. Upon these baselines, we further propose a radical-based neural network model to identify the boundary of the sensory word, and to jointly detect the original and synesthetic sensory modalities for the word. Through extensive experiments, we observe that the importance of the proposed task and dataset can be verified by the statistics and progressive performances. In addition, our proposed model achieves state-of-the-art results on the synesthesia dataset.
Introduction
Synesthesia is a linguistic and cognitive phenomenon where one sensory modality is described using concepts from another—for example, a "sweet voice" (Taste to Hearing) or "bitter cold" (Taste to Touch). While extensively studied in linguistics and neurology, it has been largely overlooked in Natural Language Processing (NLP), often conflated with general metaphor detection. However, synesthesia is distinct because it involves specific mappings between the five senses: Touch, Taste, Smell, Vision, and Hearing.
Detecting synesthesia in Chinese presents unique challenges. The language lacks word delimiters, and significant semantic information is encoded in radicals—sub-character components that often hint at a word's sensory origin. For instance, the character for "sweet" (甜) contains the "tongue" radical (舌).
Methods: Radical-Based Detection
To address these challenges, we propose a novel neural architecture that integrates sub-character information with pre-trained language models.
The framework consists of three primary components:
- Radical-Based Representation: The model incorporates radical information by mapping characters to their radicals using the Xinhua dictionary. This provides explicit sensory clues that standard character-level BERT models might miss.
- Boundary Detection: Sensory word extraction is treated as a Machine Reading Comprehension (MRC) task. The model predicts the start and end indices of the sensory word within a sentence.
- Joint Modality Detection: Once the sensory word is identified, a joint learning module predicts both the original sensory modality (the word's literal sense) and the synesthetic sensory modality (the sense it describes in context).
Dataset Construction
We constructed a new benchmark for Chinese synesthesia detection using the Sinica Corpus, a 10-million-token balanced corpus of Modern Chinese. The final dataset includes 7,217 human-annotated synesthetic sentences featuring 187 unique sensory adjectives.
The dataset reveals that visual, tactile, and gustatory adjectives are the most frequent, supporting linguistic theories like Ullmann’s "Directionality Law," which suggests synesthetic transfers typically move from "higher embodied" senses (Touch/Taste) to "lower" ones (Vision/Hearing).
Experimental Results
Our radical-based model achieved state-of-the-art (SOTA) performance across all sub-tasks:
- Word Extraction: Achieved an F1-score of 79.0, significantly outperforming BiLSTM+CRF and standard BERT baselines.
- Modality Detection: The model reached a Weighted F1 of 79.5 for original modality and 80.1 for synesthetic modality.
- Ablation Study: Removing radical information led to a performance drop, confirming that sub-character features are critical for deep semantic understanding in Chinese.
Conclusion
This work introduces the first computational framework and large-scale dataset for Chinese synesthesia detection. By leveraging Chinese radicals, our model bridges the gap between figurative linguistics and NLP. Future research will address data imbalance in rarer sensory categories like "Smell" and explore cross-lingual synesthetic patterns.
Summary updated