A study of conceptual language similarity: comparison and evaluation

Haotian Ye, Yihong Liu, Hinrich SchützearXiv 2023 · DOI · arXiv

Abstract

This study evaluates "conceptual language similarity"—a measure of how different languages categorize and associate basic concepts—against traditional lexical and typological metrics. By aligning concepts across a massive parallel corpus of the Bible, the researchers compare conceptual vectors against established databases like ASJP, URIEL, and Grambank. While conceptual similarity provides a unique, complementary perspective on language relatedness, it currently lags behind traditional methods in predicting genealogical language families, largely due to inherent biases in the alignment process toward Indo-European languages.

Introduction: Mapping the Mind of Language

In Natural Language Processing (NLP), language similarity is typically measured through "surface" features: lexical similarity (shared word roots) or typological similarity (shared grammatical rules). This research explores a third dimension: conceptual similarity. Instead of focusing on how a language speaks, it investigates how a language thinks—specifically, how it maps the world into concepts and the associations it forms between them. For example, while European languages might not link the two, Chinese, Japanese, and Korean often associate the concept of "mouth" with "entrance."

Methodology

The researchers utilized a tool called the Conceptualizer to build their framework:

  1. Alignment: Using the Parallel Bible Corpus (PBC), they aligned 83 basic concepts across 1,335 languages using a bipartite graph.
  2. Vectorization: Each concept was represented as a 100-dimensional vector, consisting of the concept itself and its 99 most statistically correlated concepts within that specific language.
  3. Evaluation Task: Language family prediction was treated as a binary classification problem: given a language, do its $k$-nearest neighbors belong to the same genealogical family?
  4. Benchmarks: The conceptual approach was compared against lexical metrics (ASJP) and typological metrics (URIEL and Grambank).

Key Results and Comparative Performance

The evaluation revealed that lexical and typological metrics remain the gold standards for predicting language families, frequently achieving near-perfect accuracy. Conceptual similarity, particularly when measured via Cosine Similarity (CosSim), showed strong performance for certain families like Atlantic-Congo and Indo-European but struggled significantly with others, such as Sino-Tibetan.

Comparison of conceptual similarity measures (CosSim, Hamming) against lexical (ASJP) and typological (URIEL) benchmarks across various language families.

A critical finding was the "Binarization Trap." Using Hamming distance (binary 1/0 associations) performed significantly worse than Cosine similarity. Binarization appeared to amplify systemic biases, drowning out unique linguistic signals in favor of broad, often inaccurate, associations.

Grambank and Structural Similarity

The study also marked one of the first major implementations of Grambank, a comprehensive database of 195 grammatical features. Despite the challenge of "unknown" values due to data sparsity, Grambank proved highly effective, showing accuracy above 80% for its largest represented families.

Performance of Grambank-based similarity in predicting language families across different k values.

Insights and Limitations

The Indo-European Bias

A major limitation identified was the "INDO magnet." Because English served as the source language for initial concept alignment, the model became heavily biased toward Indo-European (INDO) structures. In Hamming distance tests, non-Indo-European languages frequently had INDO languages as their nearest neighbors, indicating that the alignment process itself imposes a specific conceptual framework.

Complementary Perspectives

Despite these biases, conceptual similarity captures links that genetic or structural metrics miss. For instance, Tagalog and Spanish exhibit conceptual similarities that are not explained by their distinct language families, suggesting that conceptual mapping can capture cultural or contact-based influences that traditional metrics overlook.

Conclusion and Future Work

The authors conclude that while predicting genealogical families is a useful benchmark, it may not be the most appropriate test for conceptual similarity, as language families are defined by historical descent rather than shared worldview. Future research will focus on reducing alignment bias by using non-English source languages and developing new evaluation tasks, such as cross-lingual metaphor understanding or cultural sentiment analysis, where a conceptual lens is uniquely advantageous.

Summary updated

Abstract

An interesting line of research in natural language processing (NLP) aims to incorporate linguistic typology to bridge linguistic diversity and assist the research of low-resource languages. While most works construct linguistic similarity measures based on lexical or typological features, such as word order and verbal inflection, recent work has introduced a novel approach to defining language similarity based on how they represent basic concepts, which is complementary to existing similarity measures. In this work, we study the conceptual similarity in detail and evaluate it extensively on a binary classification task.
BibTeX
@misc{ye2023study,
  author = {Haotian Ye and Yihong Liu and Hinrich Schütze},
  title = {{A study of conceptual language similarity: comparison and evaluation}},
  year = {2023},
  eprint = {2305.13401},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.13401},
}
Figure