Brandeis at VarDial 2024 DSL-ML Shared Task: Multilingual Models, Simple Baselines and Data Augmentation

Jonne Sälevä, Chester Palen-MichelNAACL 2024

TL;DR

This paper presents Brandeis University's submission to the VarDial 2024 DSL-ML Shared Task, focusing on distinguishing between closely related languages and dialects. They found that surprisingly simple baselines often performed competitively, and even outperformed, complex multilingual models on certain datasets, highlighting the importance of careful evaluation and dataset characteristics.

What Problem Does This Paper Address?

Language identification (LID) is crucial for many NLP applications, but accurately distinguishing between similar languages, dialects, and variants remains a significant challenge. This is particularly true for "low-resource" languages where data is scarce and often noisy. The VarDial workshop and its associated shared tasks aim to address these issues by providing a platform for researchers to develop and evaluate models on these difficult language identification problems. This year's task specifically focused on multi-label classification for discriminating between similar languages and varieties.

Methods

The Brandeis team explored three main approaches for the closed track of the shared task:

  • Simple Baselines: They implemented a suite of non-neural models using scikit-learn, including logistic regression, random forests, and support vector machines. These models relied on bag-of-n-grams features (both word and character level) and explored different vectorization techniques (count and TF-IDF).
  • Per-Language mBERT Fine-tuning: They fine-tuned the bert-base-multilingual-cased model independently for each language group. This approach leverages the pre-trained multilingual capabilities of BERT and adapts them to the specific nuances of each language.
  • Concatenated mBERT Fine-tuning: A single mBERT model was fine-tuned on the concatenation of all the training datasets. The goal here was to create a unified model capable of handling all language groups simultaneously.

Additionally, they experimented with data augmentation techniques (naive word replacement and dependency tree-based span replacement) and a novel sentence segmentation strategy for the BCMS dataset, which had significantly longer documents.

Datasets

The shared task utilized several datasets for different macro-language groups:

  • English, Spanish, Portuguese: These datasets were sourced from DSL-TL, which uses manually annotated labels from the Discriminating Similar Languages Corpus Collection (DSLCC).
  • French: This dataset partially came from FreCDo and DSLCC. Notably, French data had named entities masked out.
  • Bosnian, Croatian, Montenegrin, Serbian (BCMS): This dataset was derived from sources like BENCHić-lang and Twitter, leading to potentially different content characteristics compared to other datasets.

The paper provides detailed statistics on the number of documents, sentences per document, and tokens per document for each language split in Table 1.

Key Insights and Ideas

  • Baselines Punch Above Their Weight: Perhaps the most striking finding is how well simple scikit-learn baselines performed. For many languages, these models were competitive with, and sometimes even surpassed, sophisticated pre-trained multilingual models like mBERT. This suggests that for certain language identification tasks, especially with limited data, simpler models can be highly effective and more interpretable.
  • Dataset Characteristics Matter: The performance differences between models, particularly the struggle of mBERT on the BCMS dataset, point to the significant impact of dataset characteristics. The BCMS dataset's unique origin (Twitter) and longer documents likely presented challenges that even powerful models found difficult to overcome without specific adaptations.
  • Segmentation for Long Documents: For the BCMS dataset, which featured unusually long documents, segmenting them into sentences and then aggregating predictions significantly boosted performance. This highlights the importance of tailoring preprocessing steps to the specific data structure.
  • Data Augmentation's Limited Impact: Despite exploring common data augmentation techniques, the authors found they did not yield significant performance improvements for this task. This suggests that the inherent difficulty might lie more in the subtle linguistic differences than in a lack of sufficient training examples for the models to learn from.

Key Results

  • Overall Performance: The Brandeis submissions achieved the best performance on all languages as reported by the organizers.
  • Scikit-learn vs. mBERT: On the test set, simple scikit-learn models often performed surprisingly well. For BCMS, the best non-neural baseline outperformed mBERT by over 4.3 macro-F1 points. For English and Portuguese, the differences were less than 4 macro-F1 points.
  • mBERT Strengths: Per-language fine-tuned mBERT models (Run 2) generally performed best for Spanish, French, and BCMS on the development set.
  • Concatenated Model: The concatenated mBERT model (Run 3) performed best for English and Portuguese on the development set, but struggled significantly with BCMS.
  • BCMS Segmentation: Applying sentence segmentation to the BCMS dataset resulted in a dramatic improvement, with macro-F1 scores increasing by over 50 points.

Limitations and Future Work

  • BCMS Performance: The authors acknowledge the persistent underperformance of the concatenated mBERT model on BCMS, hypothesizing it might be due to drastically fewer examples for BCMS languages compared to others, and the majority class (Serbian) dominating.
  • Data Augmentation: The explored data augmentation techniques did not prove beneficial and were not included in the final submissions. Future work could explore more targeted augmentation strategies.
  • Domain Mismatch: The authors speculate that inherent noisiness in low-resource data might contribute to the observed trends, suggesting further investigation into data quality and its impact.
  • Ambiguous Examples: Models struggled with ambiguous examples across languages, particularly for English and Portuguese. Future work could focus on better handling these challenging cases.

Summary updated

BibTeX
@inproceedings{saleva2024brandeis,
  author = {Jonne Sälevä and Chester Palen-Michel},
  title = {{Brandeis at VarDial 2024 DSL-ML Shared Task: Multilingual Models, Simple Baselines and Data Augmentation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure