Profiling neural grammar induction on morphemically tokenised child-directed speech

Mila Marcheva, Theresa Biberauer, Weiwei SunNAACL 2025

TL;DR

This paper investigates how well state-of-the-art neural grammar induction models can parse child-directed speech when words are broken down into their smallest meaningful units (morphemes). They found that while standard metrics like F1 score show comparable performance, new evaluation methods reveal that high F1 scores don't necessarily mean the models are learning linguistically plausible structures for functional morphemes. This highlights a critical challenge for developing cognitively realistic AI language learners.

What Problem Does This Paper Address?

Grammar induction (GI) aims to uncover the underlying grammatical structure of a language from raw text. This is a fundamental task in computational linguistics and has implications for understanding language acquisition. However, most current GI systems treat words as atomic units, ignoring the rich information contained within their morphemes (e.g., prefixes, suffixes). This is particularly problematic when studying child language acquisition (FLA), where the salience of functional morphemes (like plural "-s" or past tense "-ed") is crucial for learning. Existing neural GI models, despite achieving state-of-the-art performance on standard benchmarks, might be missing out on this crucial morphological information, leading to potentially unplausible grammatical structures, especially for functional morphemes. The paper addresses this gap by evaluating neural GI models on morphemically tokenized child-directed speech (CDS) and introducing new evaluation metrics to assess the linguistic plausibility of their learned structures.

Methods

The authors use three state-of-the-art neural grammar induction models: Compound Probabilistic Context-Free Grammar (C-PCFG), Neural PCFG (N-PCFG), and Tensor Decomposition PCFG (TN-PCFG). They employ implementations from Yang et al. (2021b).

The core methodological innovation lies in their approach to data preparation and evaluation:

  • Morphemic Tokenization: Instead of treating words as single units, they split words into their lemma and bound functional morphemes (e.g., "walks" becomes "walk" + "-s"). This process is carefully handled to exclude exceptions and ensure linguistic accuracy. The parsed tree structures are then modified to reflect this tokenization.
  • Child-Directed Speech (CDS) Focus: They specifically use CDS data, which is argued to be more cognitively realistic for studying language acquisition.
  • Novel Evaluation Metrics: Beyond the standard sentence-level F1 score, they introduce two new metrics:
  • Depth-of-morpheme: This metric checks if a bound functional morpheme is attached at the same depth in the predicted parse tree as its corresponding lexeme. It assesses the correct hierarchical attachment.
  • Sibling-of-morpheme: This metric analyzes the immediate parent node (sibling) of the functional morpheme in the parse tree. They evaluate the semantic role labels (SRLs) of these siblings to determine if the attachment is linguistically plausible.

Datasets

The primary dataset used is the CHILDES Treebank (CHITB), which contains child-directed speech (CDS) sentences with phrase structure annotations. They use the Brown-Adam data for testing, as its annotations are considered highly verified. The remaining sentences are randomly split for training and validation. They also reference the Penn Treebank (PTB) for comparison, noting that CHITB has smaller vocabulary and shorter sentences, and is less canonical (e.g., includes unfinished sentences).

Key Insights and Ideas

  • F1 Score is Not Enough: The most striking finding is that models achieving the highest F1 scores do not necessarily produce linguistically plausible structures for functional morpheme attachment. This suggests that standard evaluation metrics can mask significant errors in how models learn to represent morphology.
  • Morphemic Tokenization Matters for Plausibility: By explicitly modeling morphemes, the models have a better chance of learning their correct syntactic roles. However, even with this advantage, the models struggle with accurate attachment.
  • Cognitive Realism in Evaluation: The paper strongly advocates for evaluation methods that align with insights from First Language Acquisition (FLA). The proposed metrics (depth-of-morpheme, sibling-of-morpheme) are designed to capture aspects of linguistic plausibility that F1 misses.
  • English as a Weakly Inflecting Language: The authors acknowledge that English, being largely isolating, presents a less challenging case for morphology than highly inflected languages. This implies that the observed difficulties might be amplified in morphologically richer languages.

Key Results

  • F1 Performance: When using a large number of non-terminals and terminals, morphemically tokenized data performs on par or better than standard tokenization across models. N-PCFG (nt8192 t16384) achieves the highest F1 score (78.56% morphemic vs. 79.01% standard). The right-branching baseline also performs comparably to neural models.
  • Depth-of-Morpheme: N-PCFG, despite its high F1, shows a significantly lower percentage of correctly attached bound functional morphemes (e.g., 55.11% for "-s") compared to TN-PCFG (95.11% for "-s") and the right-branching baseline (41.11% for "-s"). This indicates N-PCFG often attaches morphemes to incorrect parts of the sentence structure.
  • Sibling-of-Morpheme (SRL Analysis): TN-PCFG demonstrates superior performance in correctly identifying the semantic role of the sibling node for bound morphemes, particularly for verbal instances (correctly attaching "-ed" and "-ing" to verbs). N-PCFG makes errors, attaching morphemes to direct objects, indirect objects, and adjuncts.

Limitations and Future Work

  • Morphemic Tokenization Dependency: The novel evaluation metrics are inherently dependent on morphemically tokenized text, limiting their direct applicability to standardly tokenized data.
  • Binary Depth Metric: The binary nature of the depth-of-morpheme metric can sometimes obscure nuanced linguistic patterns.
  • English Focus: The study is limited to English, a weakly inflecting language. The authors suggest that applying this methodology to morphologically richer languages would provide a more rigorous test and potentially reveal greater benefits and challenges.
  • Future Work: The authors plan to extend their methodology to CDS from morphologically rich languages and to expand both the linguistic scope and experimental design.

Summary updated

BibTeX
@inproceedings{marcheva2025profiling,
  author = {Mila Marcheva and Theresa Biberauer and Weiwei Sun},
  title = {{Profiling neural grammar induction on morphemically tokenised child-directed speech}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure