Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective

Taelin Karidi, Eitan Grossman, Omri Abend

Abstract

Traditional cross-lingual NLP focuses on global alignment—mapping entire vector spaces from one language to another. However, this approach often overlooks whether individual translation equivalents truly share the same semantic meaning. This paper introduces a framework for measuring "local" alignment at the word and domain levels across 16 diverse languages. By utilizing Semantic Neighborhood Comparison (SNC) and a novel validation method based on "lexical gaps," the research demonstrates that structured domains (like time and quantity) align more consistently across cultures and that historical rates of lexical change are primary predictors of modern semantic alignment.

Introduction: Beyond Global Mapping

In cross-lingual NLP, alignment is typically treated as a geometric problem: rotating the vector space of a source language to match a target language. While effective for machine translation, this global approach ignores a fundamental cognitive question: Do words that we translate as equivalents actually "carve up" the world in the same way? For example, while the English green and French vert are translations, their boundaries on the color spectrum and their cultural connotations may differ significantly.

Measuring this local alignment is challenging because there is no established ground truth for how much two words should overlap. This research addresses this gap by providing metrics for local alignment and robust methodologies to validate them across different semantic domains.

Methodology: Semantic Neighborhood Comparison

The authors utilize the Semantic Neighborhood Comparison (SNC) framework, grounded in the intuition that if two words share a meaning, their "neighbors" (semantically similar words) should also be similar across languages. Three primary metrics were evaluated:

  1. SNC-STATIC: Employs static embeddings (fastText) to find the nearest neighbors of a word in one language, translates them, and measures the correlation between distance vectors in both spaces.
  2. SNC-AVE: Extends the framework to contextualized embeddings (mBERT) by averaging representations across all layers to create a single vector per word.
  3. SNC-CLOUD (Novel): Represents a word as a "point cloud" of up to 1,000 instances from different sentences. By measuring the distance between these point clouds, the metric captures nuanced, context-dependent usage that single-vector representations miss.

To ensure these metrics reflect linguistic reality rather than model artifacts, the authors employed Synthetic Validation (shuffling neighbor indices) and Naturalistic Validation (comparing results against known "lexical gaps" in kinship terms).

Data and Validation

The study leveraged several key datasets:

  • NorthEuraLex (NEL): A lexical resource for 1,016 concepts across 107 languages, providing the basis for semantic domains.
  • Leipzig Corpus: Used to extract sentence contexts for the SNC-CLOUD metric.
  • KinDiv: A resource containing 37,370 lexical gaps in kinship terms across 699 languages, used for naturalistic validation.
  • MultiPic & TransSim: Human-evaluated datasets used to correlate automated metrics with human behavioral data.

Key Insights

1. Domain Stability and Structure

One of the paper's most significant findings is that alignment is far more consistent at the domain level than the word level. While different metrics might disagree on the alignment of a specific word like "soft," they show high consensus ($r = 0.93$) on which categories of words align best. Highly structured domains—those governed by logic or biological necessity, such as Quantity (numbers), Time, and Kinship—show the highest cross-lingual alignment. Conversely, abstract or action-oriented domains like Motion align poorly.

2. The "Rate of Change" Predictor

The research identifies a striking historical correlation: words that have undergone rapid historical change (high rate of lexical change) are significantly less aligned across modern languages. This suggests that "semantic drift" over centuries leaves a measurable footprint on the modern ability to align concepts across languages.

3. Culture vs. Geography

When examining why certain languages align better than others, the authors found that cultural distance (shared social traits, religion, etc.) is a stronger predictor of semantic misalignment than simple geographic proximity. This reinforces the idea that language is a reflection of shared cultural conceptualization rather than just physical location.

Results and Limitations

The study confirmed that contextualized metrics (SNC-AVE and SNC-CLOUD) outperform static methods in correlating with naturalistic lexical gaps. Furthermore, polysemy (a word having multiple meanings) was identified as a major driver of misalignment. For example, the English word "bank" creates neighborhood mismatches when compared to languages that distinguish between a river bank and a financial institution.

While the framework is robust, it currently faces limitations. The naturalistic validation is primarily focused on the kinship domain, and the metrics have yet to be adapted for modern autoregressive models like GPT-4. Future work aims to expand validation to domains like color and emotion and to integrate the framework with larger, more complex language models.

Summary updated

BibTeX
@inproceedings{karidi2024locally,
  author = {Taelin Karidi and Eitan Grossman and Omri Abend},
  title = {{Locally Measuring Cross-lingual Lexical Alignment: A Domain and Word Level Perspective}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.932},
}
Figure