The Effect of Data Partitioning Strategy on Model Generalizability: A Case Study of Morphological Segmentation

Zoey Liu, Bonnie J. DorrNAACL 2024 · DOI

TL;DR

This paper investigates how different ways of splitting data for training and testing affect how well NLP models generalize, using morphological segmentation across 19 diverse languages. They found that random data splits generally lead to more reliable model performance estimates and rankings compared to adversarial splits, especially when dealing with challenging, unseen data.

What Problem Does This Paper Address?

Evaluating NLP models is crucial, but the way data is split into training, validation, and test sets can significantly impact how we perceive a model's performance and its ability to generalize to new, unseen data. Prior research has shown that using a single, fixed data split can be misleading, as model performance and rankings can change drastically with different splits or even entirely new datasets. This is particularly problematic for low-resource languages or languages with complex linguistic features, where data is scarce and standard evaluation practices might not be sufficient. The authors aim to address this by systematically studying the impact of different data partitioning strategies on model generalizability across a linguistically diverse set of languages, focusing on the task of morphological segmentation.

Methods

The core of this study lies in its experimental design, which compares two primary data partitioning strategies: random splits and adversarial splits.

  • Random Splits: The standard approach where data is divided into training, validation, and test sets without any specific criteria beyond random chance.
  • Adversarial Splits: This strategy aims to maximize the "distance" or difference between the training/validation data and the test data, often by minimizing the Wasserstein distance between their distributions. The goal is to create test data that is as challenging and dissimilar as possible to the training data, simulating a more realistic "out-of-domain" scenario.

The researchers constructed new test samples of varying sizes (10-50% of the original dataset) for each language. The remaining data (residual data) was then split into training and evaluation sets using both random and adversarial strategies at a 9:1 ratio. This process was repeated multiple times to ensure robustness.

Four model architectures were used:

  • First-order Conditional Random Field (CRF): A traditional, discriminative model for sequence tagging.
  • Neural Sequence-to-Sequence (seq2seq) models: Including LSTM, Transformer, and a smaller Transformer variant (Transformer_TINY).

Model performance was evaluated using the F1 score.

Datasets

The study utilized morphological segmentation data from 19 languages, spanning 10 language families. This diverse set includes:

  • Ten indigenous or endangered languages from Mexico (Mexicanero, Nahuatl, Yorem Nokki, Wixarika, Raramuri, Popoluca, Tepehua) and South America (Shipibo-Konibo).
  • Indo-European languages (English, German).
  • Agglutinative languages (Finnish, Turkish, Indonesian, Zulu, Akan, Swahili, Telugu).

The data was sourced from various prior works and archives, with specific mention of the Archive of Indigenous Languages and the DARPA Low Resource Languages for Emerging Incidents (LORELEI) Program. The study did not introduce a new benchmark dataset but rather a novel experimental setup for evaluating existing datasets.

Key Insights and Ideas

  • Random Splits Offer More Reliable Insights: The most striking finding is that random data splits, despite being simpler, tend to provide more consistent and reliable estimates of model performance and rankings, especially when evaluating against new, unseen test data.
  • Adversarial Splits Can Be Misleading: While adversarial splits are designed to be challenging, they can sometimes lead to an overestimation of model performance on the evaluation sets, which then doesn't translate well to actual new test data. This suggests that models trained on adversarial splits might appear to perform better on the evaluation set but struggle more when faced with truly novel data.
  • Language Diversity Matters: The study's strength lies in its broad linguistic coverage. The findings generally hold across different language families and morphological types (polysynthetic, fusional, agglutinative), suggesting that the observed effects of data partitioning are not language-specific.
  • Model Rankings Can Fluctuate: The paper highlights that model rankings can shift depending on the data partitioning strategy. Random splits tend to produce more stable rankings that generalize better to new test samples.
  • Challenging Data Favors Random Splits: When test data is particularly dissimilar to training data (akin to "out-of-domain" scenarios), random splits appear to be more beneficial for achieving consistent results.

Key Results

  • Individual Model Performance: When new test samples were generated randomly, adversarial splits led to larger score differences between evaluation and test sets for all four models, indicating poorer generalization. Random splits showed minimal score differences. When new test samples were generated adversarially, adversarial splits of the residual data led to smaller score differences, but random splits still yielded better average model performance on the new test samples.
  • Model Ranking Consistency: Random splits showed higher consistency between evaluation set rankings and new test sample rankings (91.47%) compared to adversarial splits (90.26%) when new test samples were generated randomly. This consistency gap widened when new test samples were generated adversarially.
  • Statistical Significance: Regression analysis confirmed that random splits had a statistically significant positive impact on F1 scores for 16 out of 19 languages, supporting the superiority of random splits.

Limitations and Future Work

  • Data Domain Limitation: For all languages, the experimental data came from the same domain due to limited availability of datasets for morphological segmentation. Ideally, future work would explore data from different domains to better simulate real-world scenarios.
  • No Validation Set: To ensure consistency across languages with varying data availability, a separate validation (tuning) set was not used. The authors acknowledge that the need for parameter tuning itself can be an indicator of a model's potential generalization issues.
  • Task Specificity: The findings are specific to morphological segmentation. Future work should expand to other NLP tasks and investigate how data partitioning strategies perform under varying degrees of data availability.
  • Heuristic Splits: While not explored in depth due to implementation challenges, the authors suggest that heuristic splits could be valuable for other tasks (e.g., text classification) where data can be split based on specific features.

Summary updated

BibTeX
@inproceedings{liu2024effect,
  author = {Zoey Liu and Bonnie J. Dorr},
  title = {{The Effect of Data Partitioning Strategy on Model Generalizability: A Case Study of Morphological Segmentation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.157},
}
Figure