Abstract
This study evaluates "conceptual language similarity"—a measure of how different languages categorize and associate basic concepts—against traditional lexical and typological metrics. By aligning concepts across a massive parallel corpus of the Bible, the researchers compare conceptual vectors against established databases like ASJP, URIEL, and Grambank. While conceptual similarity provides a unique, complementary perspective on language relatedness, it currently lags behind traditional methods in predicting genealogical language families, largely due to inherent biases in the alignment process toward Indo-European languages.
Introduction: Mapping the Mind of Language
In Natural Language Processing (NLP), language similarity is typically measured through "surface" features: lexical similarity (shared word roots) or typological similarity (shared grammatical rules). This research explores a third dimension: conceptual similarity. Instead of focusing on how a language speaks, it investigates how a language thinks—specifically, how it maps the world into concepts and the associations it forms between them. For example, while European languages might not link the two, Chinese, Japanese, and Korean often associate the concept of "mouth" with "entrance."
Methodology
The researchers utilized a tool called the Conceptualizer to build their framework:
- Alignment: Using the Parallel Bible Corpus (PBC), they aligned 83 basic concepts across 1,335 languages using a bipartite graph.
- Vectorization: Each concept was represented as a 100-dimensional vector, consisting of the concept itself and its 99 most statistically correlated concepts within that specific language.
- Evaluation Task: Language family prediction was treated as a binary classification problem: given a language, do its $k$-nearest neighbors belong to the same genealogical family?
- Benchmarks: The conceptual approach was compared against lexical metrics (ASJP) and typological metrics (URIEL and Grambank).
Key Results and Comparative Performance
The evaluation revealed that lexical and typological metrics remain the gold standards for predicting language families, frequently achieving near-perfect accuracy. Conceptual similarity, particularly when measured via Cosine Similarity (CosSim), showed strong performance for certain families like Atlantic-Congo and Indo-European but struggled significantly with others, such as Sino-Tibetan.
A critical finding was the "Binarization Trap." Using Hamming distance (binary 1/0 associations) performed significantly worse than Cosine similarity. Binarization appeared to amplify systemic biases, drowning out unique linguistic signals in favor of broad, often inaccurate, associations.
Grambank and Structural Similarity
The study also marked one of the first major implementations of Grambank, a comprehensive database of 195 grammatical features. Despite the challenge of "unknown" values due to data sparsity, Grambank proved highly effective, showing accuracy above 80% for its largest represented families.
Insights and Limitations
The Indo-European Bias
A major limitation identified was the "INDO magnet." Because English served as the source language for initial concept alignment, the model became heavily biased toward Indo-European (INDO) structures. In Hamming distance tests, non-Indo-European languages frequently had INDO languages as their nearest neighbors, indicating that the alignment process itself imposes a specific conceptual framework.
Complementary Perspectives
Despite these biases, conceptual similarity captures links that genetic or structural metrics miss. For instance, Tagalog and Spanish exhibit conceptual similarities that are not explained by their distinct language families, suggesting that conceptual mapping can capture cultural or contact-based influences that traditional metrics overlook.
Conclusion and Future Work
The authors conclude that while predicting genealogical families is a useful benchmark, it may not be the most appropriate test for conceptual similarity, as language families are defined by historical descent rather than shared worldview. Future research will focus on reducing alignment bias by using non-English source languages and developing new evaluation tasks, such as cross-lingual metaphor understanding or cultural sentiment analysis, where a conceptual lens is uniquely advantageous.
Summary updated