The interplay between language similarity and script on a novel multi-layer Algerian dialect corpus

Samia Touileb, Jeremy Barnes

TL;DR

This paper introduces a new multi-layer corpus for the Algerian dialect, featuring parallel annotations in both Latin (NArabizi) and Arabic scripts. By benchmarking cross-lingual transfer, the authors reveal a fascinating "tug-of-war" between language similarity and script: while part-of-speech tagging is highly sensitive to both, sentiment analysis is surprisingly robust to script changes.

What Problem Does This Paper Address?

Algerian is a "low-resource" vernacular language with no standardized spelling. It is a linguistic melting pot, characterized by heavy code-switching (mixing languages like French, Arabic, and Berber) and the use of two distinct scripts: NArabizi (Arabic written with Latin characters and digits) and DZ (Algerian written in Arabic script).

For researchers building NLP tools, this presents a major challenge: Should we transfer knowledge from a language that looks like the target (same script) or one that acts like the target (similar grammar/typology)? Most cross-lingual transfer research focuses on one or the other, but rarely explores the interplay between the two in a controlled way.

Methods

The researchers used multilingual BERT (mBERT) as their primary engine for two supervised tasks: Part-of-Speech (POS) tagging and Sentiment Analysis (SA).

To isolate the effects of script versus typology, they compared transfer performance from three groups of source languages to Algerian:

  1. Script Group (Same script, different typology): Persian and Urdu (use Arabic script but are Indo-European).
  2. Typology Group (Different script, same typology): Hebrew and Maltese (Semitic languages like Algerian, but written in Hebrew or Latin scripts).
  3. Both (Same script and typology): Modern Standard Arabic (MSA).

They also performed "zero-shot" experiments, training on one language and testing on Algerian without any Algerian-specific training data, to see which linguistic features the models prioritized.

Datasets

The study builds upon the existing NArabizi treebank (approx. 1,500 sentences). The authors added several new layers of annotation:

  • Transliteration: Every NArabizi token was manually transliterated into Arabic script (DZ) and a code-switched version.
  • Sentiment: Sentences were labeled as Positive, Negative, Neutral, or Mixed.
  • Topic: Sentences were categorized into six topics (Politics, Sport, Religion, etc.).

This resulted in a unique parallel corpus where the same underlying Algerian sentences can be studied across different writing systems.

Table 1: Lexical variations of the word “Algeria” in Algerian written in NArabizi and DZ scripts.

Key Insights and Ideas

  • The Tokenization Tax: One of the most striking findings was how mBERT handles different scripts. As shown in Figure 1, mBERT shatters NArabizi words into many more "subtokens" compared to the same words written in Arabic script (DZ). Because mBERT saw much more Arabic script during its initial training, it "understands" the structure of DZ words better, even if it hasn't seen that specific dialect before.
Figure 1: The effect of mBERT tokenization on MSA, Algerian Arabic (DZ), and NArabizi.
  • Task-Specific Sensitivity: The "interplay" isn't universal. POS tagging—a structural, syntactic task—is deeply affected by typology. However, Sentiment Analysis—a more semantic, "vibes-based" task—is much more resilient to script and typological differences.
  • Typology Trumps Script for Grammar: For POS tagging, transferring from Maltese (same typology, different script) actually worked better than transferring from Persian or Urdu (same script, different typology). This suggests that for grammatical tasks, the underlying "DNA" of the language matters more than the "clothes" (script) it wears.

Key Results

  • POS Tagging: The best zero-shot transfer for NArabizi came from Maltese (37.8% accuracy) and Hebrew (32.7%), confirming the importance of typological similarity. Interestingly, even though mBERT prefers Arabic script, MSA (Modern Standard Arabic) performed poorly on NArabizi (20.0%) because the script mismatch was too great.
  • Sentiment Analysis: Results were much more stable. Transferring from MSA to NArabizi yielded a Macro F1 of 62.4, which is remarkably close to the in-language performance (72.1).
  • Monolingual Baseline: When training and testing on the same script, the model performed significantly better on DZ (82.5% POS accuracy) than on NArabizi (76.3%), likely due to the more efficient tokenization mentioned above.

Limitations and Future Work

The authors acknowledge that the dataset is still relatively small (1,500 sentences), which can make neural models unstable. Furthermore, while they corrected some errors in the original treebank, the non-standardized nature of Algerian means that "correct" spelling is often a moving target.

Future work aims to dive deeper into the code-switching aspect—specifically how to handle sentences that flip between scripts and languages mid-stream—and to expand the corpus to enable more robust syntactic analysis.

Summary updated

Abstract

Recent years have seen a rise in interest for cross-lingual transfer between languages with similar typology, and between languages of various scripts. However, the interplay between language similarity and difference in script on cross-lingual transfer is a less studied problem. We explore this interplay on cross-lingual transfer for two supervised tasks, namely part-of-speech tagging and sentiment analysis. We introduce a newly annotated corpus of Algerian user-generated comments comprising parallel annotations of Algerian written in Latin, Arabic, and code-switched scripts, as well as annotations for sentiment and topic categories. We perform baseline experiments by fine-tuning multi-lingual language models. We further explore the effect of script vs. language similarity in cross-lingual transfer by fine-tuning multi-lingual models on languages which are a) typologically distinct, but use the same script, b) typologically similar, but use a distinct script, or c) are typologically similar and use the same script. We find there is a delicate relationship between script and typology for part-of-speech, while sentiment analysis is less sensitive.
BibTeX
@inproceedings{touileb2021interplay,
  author = {Samia Touileb and Jeremy Barnes},
  title = {{The interplay between language similarity and script on a novel multi-layer Algerian dialect corpus}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure