A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets

Tanja Samardzic, Ximena Gutierrez, Christian Bentz, Steven Moran, Olga PelloniNAACL 2024 · DOI

TL;DR

This paper introduces a novel, transparent method for measuring linguistic diversity in NLP datasets, moving beyond simple language counts. By representing languages as sets of features and using a Jaccard-based similarity score, they can identify specific linguistic phenomena that are underrepresented, offering a more nuanced understanding of dataset quality and guiding future data collection efforts.

What Problem Does This Paper Address?

As NLP models increasingly aim for multilingual capabilities, the creation of diverse datasets is crucial for ensuring models generalize well across languages. However, current methods for assessing this diversity often rely on simple counts of languages or language families, which fail to capture the structural differences between languages. This oversight can lead to datasets that, despite having many languages, might still be biased towards certain linguistic structures, leaving others underrepresented. The authors aim to address this by proposing a method that quantifies linguistic diversity based on actual linguistic features, allowing for a more precise understanding of what makes a dataset truly diverse and what might be missing.

Methods

The core of the paper is a new measure for linguistic diversity, the Jaccard minmax similarity (Jmm). This method treats languages as points in a feature space and compares the distribution of these points in a given dataset against a reference distribution.

Here's a breakdown:

  • Feature Representation: Languages are represented by sets of features. These features come from two main sources:
  • Grammar Features: Extracted from typological databases like WALS (World Atlas of Language Structures). These capture structural properties of languages (e.g., word order, morphology).
  • Text Features: Derived from statistical properties of text, specifically the mean word length. The authors argue this is a good proxy for morphological complexity, as longer words often indicate richer morphology.
  • Jaccard Minmax Similarity (Jmm): This is a generalized version of the Jaccard index, adapted for comparing distributions of numerical attributes (features). It quantifies the overlap between two sets of measurements (e.g., feature values for languages in two different datasets). The score ranges from 0 (no overlap) to 1 (perfect overlap), with higher scores indicating greater similarity and, by extension, less diversity relative to the reference.
  • Reference Dataset: The authors use the TeDDi sample as their reference, which is designed to maximize linguistic diversity across both grammar and text features.

Datasets

The paper evaluates several popular multilingual NLP datasets. The primary reference dataset used for comparison is the TeDDi sample.

The datasets analyzed include:

  • Universal Dependencies (UD)
  • Bible100
  • mBERT
  • XTREME
  • XGLUE
  • XNLI
  • XCOPA
  • TyDiQA
  • XQuAD

Key Insights and Ideas

  • Beyond Language Counts: The most significant insight is that simply having many languages or language families in a dataset does not guarantee high linguistic diversity. Structural differences matter, and current datasets often overlook this.
  • Mean Word Length as a Proxy: The paper demonstrates that mean word length, a text-based feature, is a surprisingly good and easily computable proxy for morphological complexity. This is crucial because obtaining detailed morphological features from typological databases is often hampered by data sparsity and requires significant human expertise.
  • Transparency in Diversity Assessment: The Jmm score is valuable because it's interpretable. It doesn't just give a number; it visually shows which linguistic features are underrepresented in a dataset compared to a reference. This allows researchers to pinpoint specific areas for improvement.
  • Underrepresented Language Types: A consistent finding across many datasets is the underrepresentation of languages with rich morphology, often indicated by longer word lengths. Many endangered languages fall into this category.
  • Consistency Across Measures: The proposed Jmm score shows good consistency with existing measures like the Typological Index (TI), validating its effectiveness.

Key Results

  • The TeDDi sample consistently scores highest in linguistic diversity across both syntactic and morphological features, confirming its role as a strong reference.
  • Many popular multilingual datasets, including mBERT and XTREME, show moderate to low morphological diversity.
  • Datasets like XCOPA and TyDiQA, despite having fewer languages, achieve higher diversity scores than larger datasets like mBERT, highlighting the importance of feature coverage over sheer language count.
  • A strong correlation (p = 0.69) was found between mean word length and a morphological complexity score (CWALS), supporting the use of mean word length as a text-based feature.
  • The study identified that languages with rich morphology (often associated with longer words) are consistently missing or underrepresented in most analyzed NLP datasets.

Limitations and Future Work

  • Feature Representation Mismatch: A limitation noted is the use of different types of features for syntax (expert-derived) and morphology (text-derived). The authors suggest that future work could explore text-based syntactic features to create a more unified approach.
  • Approximation of Complexity: While mean word length is a good proxy for morphological complexity, it's still an approximation. The correlation with CWALS is strong but not perfect, indicating that more sophisticated measures might be needed for finer-grained analysis.
  • Finding Underrepresented Languages: While the Jmm score identifies missing language types, finding specific languages that fit these criteria (especially low-resource ones) remains a challenge. The authors suggest that future work could focus on developing methods to precisely identify and incorporate such languages.
  • Automated Syntactic Text Features: The paper highlights the need for developing text-based features for syntax, similar to how mean word length serves for morphology, to enable fully automated and transparent diversity assessment.

Summary updated

BibTeX
@inproceedings{samardzic2024measure,
  author = {Tanja Samardzic and Ximena Gutierrez and Christian Bentz and Steven Moran and Olga Pelloni},
  title = {{A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.213},
}
Figure