Disambiguation of morpho-syntactic features of African American English – the case of habitual be

Harrison Santiago, Joshua Martin, Sarah Moeller, Kevin TangACL 2022

TL;DR

This paper tackles the challenge of bias in NLP systems against African American English (AAE) by focusing on the disambiguation of "habitual be." They introduce a pipeline combining rule-based filtering and data augmentation to create a balanced dataset, leading to significantly improved classification performance for this AAE-specific feature.

What Problem Does This Paper Address?

Natural Language Processing (NLP) systems often exhibit biases against speakers of African American English (AAE). This bias stems from two main issues: (1) AAE possesses unique linguistic features not well-represented in standard training data, and (2) these features often occur with low probability, making it difficult for models to learn them. This paper specifically addresses the challenge of disambiguating "habitual be," a morphosyntactic feature in AAE that indicates habitual action. The difficulty arises because "habitual be" is morphologically identical to other uses of "be" in English, and it's relatively rare even within AAE corpora, leading to imbalanced datasets that favor standard English features.

Methods

The authors propose a novel pipeline to overcome the data scarcity and ambiguity of "habitual be":

  • Preprocessing: Text segments are standardized, and non-speech elements are removed.
  • Rule-Based Filtering: A set of syntactic rules, based on the part-of-speech of words surrounding "be," is used to identify and remove a significant portion of non-habitual "be" instances. This step aims to eliminate "easy" cases of non-habitual "be" to focus the classifier on more challenging instances.
  • Data Augmentation: To increase the proportion of habitual "be" instances, they employ data augmentation techniques using the nlpaug library. This involves substituting and inserting words using Word2Vec and WordNet to create synthetic examples of habitual "be," thereby balancing the dataset.
  • Classification: The filtered and augmented data is then used to train three classifiers: logistic regression, a multi-layer perceptron (MLP), and a linear Support Vector Machine (SVM). An ensemble model combining these classifiers is also evaluated.

Datasets

The primary dataset used is the Corpus of Regional African American Language (CORAAL). This corpus contains over 150 sociolinguistic interviews with African American speakers. For this study, 5,133 instances of "be" were manually annotated as habitual or non-habitual. This resulted in an initial imbalance of 477 habitual "be" instances and 4,656 non-habitual "be" instances.

Key Insights and Ideas

  • Syntactic context is crucial: The paper reinforces that the surrounding words and their grammatical roles are strong indicators of whether "be" is used habitually. This highlights the importance of feature engineering that captures syntactic relationships.
  • Rule-based filtering is effective for easy cases: By removing clear instances of non-habitual "be" using syntactic rules, the problem is simplified, allowing machine learning models to focus on more ambiguous cases.
  • Data augmentation can balance rare features: Techniques like Word2Vec and WordNet augmentation are shown to be effective in creating synthetic data for rare linguistic features, making it possible to train models on imbalanced datasets.
  • Ensemble models offer robustness: Combining multiple classifiers (logistic regression, SVM, MLP) in an ensemble approach yielded the best performance, suggesting that diverse model perspectives improve overall accuracy and reduce variability.

Key Results

The proposed pipeline significantly improved the classification of habitual "be":

  • The rule-based filter reduced the number of non-habitual "be" instances from 4,656 to 994, increasing the proportion of habitual "be" from 9% to 30% in the filtered corpus.
  • After augmentation, the proportion of habitual "be" reached 50%.
  • The ensemble model achieved an average F1-score of 0.65 on the task of disambiguating habitual "be" after filtering and augmentation. This is a substantial improvement compared to models trained on the original, imbalanced data, where F1-scores were considerably lower (e.g., 0.416 for logistic regression, 0.498 for SVM, 0.542 for MLP without augmentation).
  • The variability in classifier performance also decreased after filtering and augmentation, indicating more stable and reliable predictions.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Filter error rate: The rule-based filter incorrectly removed 61 instances of habitual "be," indicating a potential error rate of about 13% that could be improved with more ad-hoc rules.
  • Limited context features: While POS tags of surrounding words were used, the full surrounding words or their embeddings were not included due to the risk of sparse word vectors with limited data. Future work could explore incorporating these if more data becomes available.
  • Geographic and regional variations: The study treated the CORAAL corpus as a whole and did not account for geographic location or speaker origin. Further analysis of performance across regional sub-varieties of AAE is suggested.
  • Comparison with advanced models: The authors plan to compare their results with state-of-the-art neural models like Transformers, especially in low-resource settings.
  • Generalizability to other AAE features: The pipeline is expected to be adaptable to other unique AAE features, such as the completive "done."

Summary updated

BibTeX
@inproceedings{santiago2022disambiguation,
  author = {Harrison Santiago and Joshua Martin and Sarah Moeller and Kevin Tang},
  title = {{Disambiguation of morpho-syntactic features of African American English – the case of habitual be}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure