Linear Discriminative Learning: a competitive non-neural baseline for morphological inflection

Cheonkam Jeong, Dominic Schmitz, Akhilesh Kakolu Ramarao, Anna Stein, Kevin TangACL 2023

TL;DR

This paper introduces Linear Discriminative Learning (LDL) as a competitive non-neural baseline for modeling morphological inflection in Korean. The authors demonstrate that LDL achieves high accuracy, comparable to transformer models, while offering greater interpretability and computational efficiency, making it a promising tool for future research in this area.

What Problem Does This Paper Address?

The field of computational linguistics often grapples with modeling human language acquisition and processing. Specifically, understanding how humans learn and apply morphological rules (like verb conjugations or noun plurals) is a key challenge. While deep learning models, particularly transformers, have achieved state-of-the-art results in many NLP tasks, they often lack interpretability and can be computationally expensive. This paper addresses the need for models that are not only accurate but also cognitively plausible, interpretable, and efficient, especially for tasks like morphological inflection where understanding the underlying linguistic mechanisms is crucial. The authors aim to provide a strong non-neural baseline that can compete with neural approaches in this domain.

Methods

The core of this paper is the application and evaluation of Linear Discriminative Learning (LDL) for morphological inflection in Korean. LDL is a symbolic, rule-based approach that models the relationship between form (e.g., sounds or spellings) and meaning using linear mappings. Unlike deep learning models with complex, multi-layered architectures, LDL uses simple, two-layer networks, making it highly interpretable.

The authors implemented two main approaches:

  1. Linear Discriminative Learning (LDL): This model learns a linear mapping between form (represented by triphones, sequences of three phonemes) and meaning (represented by morphemes). They conducted two studies with LDL:
  2. Study 1: Trained on combined corpus data to predict experimental data. This study faced challenges due to unseen triphones in the experimental data.
  3. Study 2: Trained individual LDL networks for each participant using both corpus and experimental data, then used these to predict other participants' productions. This approach proved more successful.
  4. Transformer Model: A standard encoder-decoder transformer architecture was implemented for comparison, representing a state-of-the-art neural approach.

Data preprocessing was a significant step, involving manual correction of inconsistencies in transcriptions and unification of styles across different corpora.

Datasets

The study utilized the following datasets:

  • NIKL Korean Dialogue Corpus (NIKL, 2021): A large adult corpus of Korean speech, annotated for morphological boundaries and phonological processes like Post-Obstruent Tensification (POT) and Consonant Simplification (CS).
  • Ko Corpus: A child spontaneous speech corpus, also in Romanized form, providing target words and their productions.
  • Experimental Data: Responses from 12 participants (plus 4 for development) were collected, including trial information, participant ID, language familiarity, and word frequency.

Key Insights and Ideas

  • LDL as a Strong Non-Neural Baseline: The paper convincingly demonstrates that LDL can achieve performance competitive with transformer models on morphological inflection tasks, challenging the notion that neural models are always superior.
  • Interpretability Matters: LDL's simple, linear architecture makes it highly interpretable, allowing researchers to understand why a model makes certain predictions, which is often difficult with deep learning models.
  • Importance of Experimental Data: Training solely on corpus data was insufficient for accurate prediction of experimental items. Incorporating individual participants' experimental data was crucial for achieving high performance.
  • Cognitive Plausibility: The authors highlight LDL's alignment with cognitive theories, such as the Rescorla-Wagner Rule, making it a more appealing choice for cognitive research compared to opaque neural models.
  • Efficiency: LDL models are computationally efficient, requiring significantly less processing time than transformer models.

Key Results

  • The best performing LDL model achieved an overall accuracy of 83.32% on the SIGMORPHON 2023 task 2 for Korean morphological inflection.
  • This accuracy was comparable to, and in some aspects better than, the transformer model.
  • The LDL model performed consistently across different item types (low-frequency, high-frequency, and pseudowords), whereas the transformer model showed more sensitivity to item types.
  • One LDL model required approximately 35 seconds of CPU processing, highlighting its speed and efficiency.

Limitations and Future Work

  • Limited Training Data: The SIGMORPHON shared task provided a relatively small amount of training data, which poses a challenge for models that typically require large datasets.
  • Data Augmentation: The authors did not employ data augmentation techniques, which could potentially improve model performance.
  • Speaker-Specific Information: While the study touched upon speaker variability, they did not fully explore or optimize the use of speaker-specific information, particularly for the transformer models.
  • Token vs. Type Frequency: The study assumed that morphological patterns are based on type frequency rather than token frequency, which might be a simplification.
  • Unattested Triphones: The initial LDL model (Study 1) struggled with experimental items containing triphones not present in the training corpus. The authors suggest using phonological features could mitigate this.
  • Future Work: The authors suggest future research could involve training models on all possible feature combinations, exploring speaker-specific information more deeply, and experimenting with training on different item frequency categories (high, low, pseudoword) separately. They also propose LDL as a valuable non-neural baseline for future shared tasks.

Summary updated

BibTeX
@inproceedings{jeong2023linear,
  author = {Cheonkam Jeong and Dominic Schmitz and Akhilesh Kakolu Ramarao and Anna Stein and Kevin Tang},
  title = {{Linear Discriminative Learning: a competitive non-neural baseline for morphological inflection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure