Shallow Parsing for Nepal Bhasa Complement Clauses

Borui Zhang, Abe Kazemzadeh, Brian ReeseACL 2022

TL;DR

This paper tackles the challenge of annotating endangered languages by developing a shallow parsing model for Nepal Bhasa complement clauses. They demonstrate that even with limited annotated data, NLP techniques like transfer learning with pre-trained models can achieve surprisingly good performance, paving the way for faster linguistic fieldwork and corpus building.

What Problem Does This Paper Address?

Documenting and analyzing endangered languages is crucial but often hampered by the slow and labor-intensive process of data collection and annotation. This is particularly true for languages with complex syntactic structures, like complement clauses, where understanding the nuances of embedded clauses and their relationship to the main verb is vital for linguistic research. Nepal Bhasa, an endangered language spoken in the Kathmandu Valley, presents such a challenge. While existing NLP tools are powerful, they often require large annotated datasets, which are scarce for low-resource languages. This paper aims to bridge this gap by exploring how to effectively train NLP models for syntactic annotation with minimal manual effort, making linguistic fieldwork more efficient.

Methods

The authors employed a shallow parsing approach, focusing on identifying and labeling complement clauses within Nepal Bhasa sentences. Their methodology involved:

  • Data Annotation: Native speaker consultants were trained to annotate raw Nepal Bhasa text, identifying embedded clauses, matrix verbs, and embedded verbs.
  • Feature Engineering: They experimented with both bigram and trigram features, considering forward and reverse directions of processing to capture syntactic dependencies.
  • Machine Learning Models: They implemented and compared three models:
  • Naive Bayes: A traditional probabilistic classifier.
  • Maximum Entropy (MaxEnt): Another statistical model that learns weights for features.
  • mBERT via NERDA: This is the core of their novel approach. They fine-tuned the pre-trained multilingual BERT model (mBERT) using the NERDA library, which is designed for low-resource sequence labeling tasks. This leverages the power of large language models without requiring extensive language-specific pre-training.

Datasets

  • OSCAR Nepal Bhasa Corpus: The primary dataset used was a subset of the Open Super-large Crawled Aggregated coRpus (OSCAR), containing approximately 5.7MB (16,694 sentences) of unannotated Nepal Bhasa data.
  • Manually Annotated Data: A small, pre-selected set of sentences from the OSCAR corpus was manually annotated by native speakers. Specifically, 200 sentences containing the complementizer 'dhakā' and 100 sentences containing 'ki' were selected for annotation, totaling 300 sentences. This represents a very small fraction (around 2%) of the available OSCAR data.

Key Insights and Ideas

  • Transfer Learning is Key for Low-Resource Languages: The success of the fine-tuned mBERT model (NERDA) highlights the immense potential of transfer learning. Pre-trained models capture general linguistic knowledge that can be effectively adapted to specific tasks and languages with minimal data.
  • Head-Final Structure Influences Prediction: The head-final nature of Nepal Bhasa (where the verb typically comes after its object and complement clauses) makes predicting the right boundary of a complement clause more accurate than the left. This is because the complementizer often appears at the right periphery of the clause.
  • Simple Annotation Schemes Can Be Effective: The paper demonstrates that even a "shallow" parsing approach, focusing on chunk identification rather than full syntactic tree parsing, can yield valuable results for linguistic fieldwork.
  • Reversing Processing Order Isn't Always Beneficial: Contrary to their hypothesis, processing sentences in a right-to-left order (mimicking head-final structure) did not improve performance, suggesting that the bidirectional nature of transformers might already handle such complexities.

Key Results

  • High Token Accuracy: The mBERT-based NERDA model achieved a token accuracy of 96.28%, significantly outperforming the other models.
  • Competitive Chunking Performance: While precision for the NERDA model was lower (62.50%) compared to some other models, its recall (76.92%) and F1 score (68.97%) were competitive, especially considering the limited training data.
  • Improved Recall with mBERT: The NERDA model showed higher recall (77% vs. 69%) compared to the maximum entropy models, indicating it was better at identifying actual complement clauses.
  • Predicting Right Edges is Easier: The NERDA model achieved 100% accuracy in predicting the right edge of complement clauses, while the left edge prediction was at 69% accuracy.

Limitations and Future Work

  • Limited Data: The authors acknowledge that the small size of the annotated dataset is a significant limitation, and model performance is heavily dependent on this limited data.
  • Inability to Handle Nested Clauses: The shallow parsing approach, by definition, cannot handle recursive or nested complement clauses. The paper notes that few such instances were found in their data, and they chose to label only the outermost CP.
  • Sentence Length Restrictions: The training algorithm discarded certain longer sentences, which might have excluded valuable data.
  • Future Directions:
    • Improving Annotator Agreement: Implementing agreement tests for annotators could enhance annotation quality.
  • Active Learning: Exploring active learning strategies to intelligently select data for annotation could further optimize the process.
  • More Complex Models: Building more sophisticated models that can generate full syntactic trees for low-resource languages.
  • Cross-Lingual Transfer: Investigating how these techniques can be applied to other endangered languages.

Summary updated

BibTeX
@inproceedings{zhang2022shallow,
  author = {Borui Zhang and Abe Kazemzadeh and Brian Reese},
  title = {{Shallow Parsing for Nepal Bhasa Complement Clauses}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure