Introducing QuBERT: A Large Monolingual Corpus and BERT Model for Southern Quechua

Rodolfo Zevallos, John Ortega, William Chen, Richard Castro, Núria Bel, Cesar Yoshikawa, Renzo Venturas, Hilario Aradiel, Nelsi MelgarejoNAACL 2022

TL;DR

This paper introduces QuBERT, a large monolingual corpus and a BERT-based language model for Southern Quechua, a low-resource indigenous language. This work addresses the critical lack of digital resources for Quechua, enabling improved performance on NLP tasks like Named Entity Recognition (NER) and Part-of-Speech (POS) tagging, with results comparable to higher-resource languages.

What Problem Does This Paper Address?

The digital landscape is heavily skewed towards high-resource languages, leaving many languages with limited online data, including indigenous languages spoken by millions. This scarcity hinders the development of essential NLP tools like machine translation, search engines, and chatbots for these communities. Southern Quechua, spoken by millions in Peru, is a prime example of such a low-resource language, lacking sufficient digital resources for robust NLP model development. This paper aims to bridge this gap by creating a substantial, publicly available dataset and a powerful pre-trained language model for Southern Quechua.

Methods

The authors introduce several key methodological contributions:

  • Corpus Curation: They compiled a large, diverse monolingual corpus of Southern Quechua, combining data from various sources, including existing corpora like Opus and JW300, but expanding significantly beyond them.
  • Normalization Technique: A normalization technique, previously used in smaller-scale Quechua NLP work, is applied to the curated corpus. This technique utilizes finite-state transducers (FSTs) to standardize text.
  • Tokenization Techniques: They explore and compare several state-of-the-art sub-word tokenization techniques:
  • Byte-Pair Encoding (BPE): A widely used baseline for agglutinative languages.
  • BPE-Guided: An extension of BPE that incorporates prior knowledge of Quechua suffixes to improve segmentation.
  • Prefix-Root-Postfix-Encoding (PRPE): A technique that explicitly separates words into prefixes, roots, and suffixes.
  • QuBERT Model: A BERT-based transformer model, specifically a RoBERTa enhancement, is pre-trained on the curated Quechua corpus. This model, named QuBERT, is then fine-tuned for downstream tasks.

Datasets

The primary dataset introduced is a large monolingual corpus for Southern Quechua, comprising approximately 450,000 segments (or 384,184 sentences and 4,408,953 tokens). This corpus is a compilation of 50 monolingual corpora from diverse sources, including:

  • JW300 (Agić and Vulić, 2019)
  • OSCAR (Suárez et al., 2019)
  • CC-100 (Conneau et al., 2020; Wenzek et al., 2020)
  • JW300_2017
  • Wikipedia_2021
  • And many others detailed in Table 4 of the paper.

Additionally, the authors created two benchmark datasets for evaluation:

  • NER Corpus: 5,450 sentences annotated for Named Entity Recognition (Person, Location, Organization).
  • POS Corpus: 4,229 sentences annotated for Part-of-Speech tagging.

Key Insights and Ideas

  • The Power of Scale for Low-Resource Languages: The paper demonstrates that even for low-resource languages, creating a large, diverse corpus can significantly boost NLP performance, making it competitive with higher-resource languages.
  • Normalization and Tokenization Matter: The choice of normalization and tokenization techniques has a noticeable impact on downstream task performance, with PRPE showing particular promise for NER.
  • Quechua's Complexity: The highly agglutinative and polysynthetic nature of Quechua presents unique challenges for NLP, requiring specialized approaches to morphology and word segmentation.
  • Public Resources are Crucial: By releasing the corpus and the QuBERT model, the authors aim to foster collaboration and accelerate research in Quechua NLP, democratizing access to these valuable resources.

Key Results

The paper reports promising results on two key NLP tasks:

  • Named Entity Recognition (NER):
    • QuBERT achieved F1 scores between 71% and 74% across different tokenization and normalization approaches.
  • The PRPE tokenization technique, when combined with normalization, yielded the highest accuracy for NER.
  • While slightly lower than state-of-the-art for high-resource languages, these scores are competitive with recent work on other low-resource languages.
  • Part-of-Speech (POS) Tagging:
    • QuBERT achieved accuracy in the high 80% range (84-87% F1 scores) for POS tagging, demonstrating strong performance.
  • PRPE outperformed other techniques in correctly classifying adjectives, while BPE-Guided showed slight advantages in determinant detection.

Limitations and Future Work

  • Limitations:
    • The authors acknowledge that the NER accuracy scores are somewhat lower than those achieved for high-resource languages.
  • While the annotators were native speakers, they were also students, suggesting that professional annotators might yield even higher quality annotations.
  • Future Work:
    • Experimenting with more complex model architectures for NER, such as those used in SemEval-2022.
  • Involving more native Quechua speakers as annotators to further improve data quality.
  • Expanding the scope to include models for machine translation, question-answering, and topic modeling, which require even richer reference data.
  • Developing online tools for native Quechua speakers and exploring human-computer interaction applications.

Summary updated

BibTeX
@inproceedings{zevallos2022introducing,
  author = {Rodolfo Zevallos and John Ortega and William Chen and Richard Castro and Núria Bel and Cesar Yoshikawa and Renzo Venturas and Hilario Aradiel and Nelsi Melgarejo},
  title = {{Introducing QuBERT: A Large Monolingual Corpus and BERT Model for Southern Quechua}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure