Mind the Gap: How BabyLMs Learn Filler-Gap Dependencies

Chi-Yun Chang, Xueyang Huang, Humaira Nasir, Shane Storks, Olawale Akingbade, Huteng DaiEMNLP 2025

TL;DR

This paper investigates whether small language models, trained on child-directed data, can learn complex grammatical structures like filler-gap dependencies, a key area in language acquisition research. The findings suggest these models partially acquire these dependencies but struggle with generalizing across constructions and respecting structural constraints like island effects, leaving the "poverty of the stimulus" debate open for AI.

What Problem Does This Paper Address?

The acquisition of complex grammatical structures by children from limited and noisy input is a long-standing puzzle in linguistics. One such structure is the "filler-gap dependency," where a phrase (like a "wh-phrase" such as "who" or "what") at the beginning of a sentence licenses a missing element (a "gap") later in the sentence. For example, in "Who did the intern say ___ left the meeting early?", "who" is the filler and the blank is the gap. These dependencies are not arbitrary; they are constrained by certain syntactic structures, known as "islands," which block the formation of such dependencies. For instance, it's ungrammatical to form a filler-gap dependency across a "wh-island" like in "*What did he leave [before she finished ___]?"

This paper asks whether modern neural language models (LMs), specifically GPT-2, can learn these complex dependencies and constraints when trained on data that mimics the input children receive. This is crucial for understanding if LMs can replicate human-like language acquisition or if innate biases are truly necessary, as suggested by the "poverty of the stimulus" argument.

Methods

The researchers employed a suite of syntactic evaluations inspired by prior work to probe the models' understanding of filler-gap dependencies. Their methodology involved:

  • GPT-2 Models: They used four GPT-2 models: two base models trained on 10 million (10M) and 100 million (100M) tokens from the BabyLM dataset, and two competitive models from the BabyLM challenge (ConcreteGPT and BabbleGPT). A standard, large-corpus-trained GPT-2 served as a high-resource benchmark.
  • Syntactic Constructions: They tested four types of constructions:
  • Gap Distance: How increasing intervening material affects dependency learning.
  • Multiple Gaps: Handling more than one gap in a sentence.
  • Wh-Islands: Testing sensitivity to wh-islands that block dependencies.
  • Adjunct Islands: Testing sensitivity to adjunct islands.
  • Evaluation Metrics:
    • Wh-Licensing Score: Measures how much a filler reduces the model's surprise at the gap position.
  • Flip Test: Assesses if the model's surprisal flips direction based on the presence or absence of a gap, indicating bijectivity.
  • Grammaticality Division Test: Compares surprisal between grammatical and ungrammatical sentences.

Datasets

  • Training Data: The primary training data came from the BabyLM Challenge datasets. These datasets are designed to mimic child-directed language input, with approximately 70% being child-oriented (e.g., CHILDES interactions, children's literature) and the remaining 30% being adult-oriented (e.g., BNC dialogue, Switchboard, OpenSubtitles). Models were trained on 10M and 100M token subsets.
  • Evaluation Data: Custom-designed test sets were created for the four syntactic constructions mentioned above, with 20 items per construction.

Key Insights and Ideas

  • Partial Acquisition, Not Mastery: Even with child-directed data, none of the models fully mastered filler-gap dependencies across all constructions. This suggests that while LMs can learn some aspects of these complex structures, they lack the robust generalization seen in human learners.
  • Data Scale Matters (to a point): The 100M token models generally outperformed the 10M token models, indicating that more data, even if child-directed, aids learning. However, even the largest models struggled with certain constraints.
  • Island Constraints are Tough: Models showed particular difficulty in respecting "island constraints," which are crucial for grammatical well-formedness. This suggests that learning these structural boundaries is a significant challenge for current LM architectures.
  • Bijectivity is Tricky: The "flip test" revealed that models often captured only one direction of the filler-gap bijectivity (e.g., understanding that a gap is expected when a filler is present, but not necessarily that a filler increases surprise when no gap is present).
  • Child-Directed Data Alone Isn't Enough: An experiment retraining models on only child-oriented data showed similar limitations, suggesting that adult-oriented data isn't the sole culprit for model failures, and that innate biases might be more critical than previously thought for human acquisition.

Key Results

  • GPT-2-10M: Showed no statistically significant acquisition of filler-gap dependencies across any construction.
  • GPT-2-100M: Acquired dependencies for gap-distance-obj (both locally and globally) and showed some robustness to intervening material length. It also demonstrated some learning of island constraints for wh-islands.
  • ConcreteGPT & BabbleGPT: These specialized BabyLM models showed slight improvements over the base 10M model, with BabbleGPT performing competently on grammaticality judgments, nearing the threshold GPT-2 model's performance. However, they still struggled with generalizing to island constraints.
  • Grammaticality: While the threshold GPT-2 model passed grammaticality tests for all constructions, the trained models showed mixed results, with BabbleGPT performing best among the BabyLM models.

Limitations and Future Work

The authors acknowledge several limitations:

  • Architecture: The study focused solely on GPT-2. Exploring other architectures (e.g., LLaMA, hybrid models) could reveal different learning capabilities.
  • Hyperparameter Tuning: The search space for hyperparameter tuning was limited by computational constraints.
  • Data Diversity: The BabyLM corpora, while child-oriented, represent a narrow slice of linguistic input and lack multimodal grounding, prosody, and rare constructions.
  • Size Mismatch: The child-only subset used in a control experiment was smaller than the full corpus, potentially confounding results.

Future work could involve exploring alternative architectures, more diverse and multimodal datasets, and investigating how inductive biases interact with data distributions to facilitate human-like language acquisition. The paper concludes that while LMs can learn some aspects of complex syntax from child-directed data, they still fall short of human-like sensitivity to structural constraints, leaving the question of innate biases versus statistical learning open.

Summary updated

BibTeX
@inproceedings{chang2025mind,
  author = {Chi-Yun Chang and Xueyang Huang and Humaira Nasir and Shane Storks and Olawale Akingbade and Huteng Dai},
  title = {{Mind the Gap: How BabyLMs Learn Filler-Gap Dependencies}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure