Semantic Training Signals Promote Hierarchical Syntactic Generalization in Transformers

Aditya Yedetore, Najoung KimEMNLP 2024 · DOI

Introduction: The Poverty of the Stimulus in Neural Networks

A central debate in linguistics, known as the "poverty of the stimulus," posits that the linguistic input children receive is too sparse and ambiguous to explain their mastery of complex hierarchical grammar rules. A classic example is yes/no question formation. Given a sentence like "The newt who is cold is sleeping," a hierarchical rule correctly moves the main auxiliary to the front: "Is the newt who is cold sleeping?" In contrast, a simpler linear rule might move the first auxiliary encountered, resulting in the ungrammatical "Is the newt who cold is sleeping?"

Hierarchical syntactic movement for question formation.

While humans intuitively grasp these hierarchical structures, standard Transformer models—the backbone of modern Large Language Models (LLMs)—often default to linear shortcuts when trained solely on text. This paper investigates whether providing models with "meaning" (structured logical representations) can bridge this gap, allowing them to discover hierarchical syntax through the relationship between form and semantics.

Methodology: Training on Form vs. Meaning

The researchers evaluated standard Transformer architectures across two primary tasks: Sequence-to-Sequence (Seq2Seq) translation and Language Modeling (LM). They compared two distinct training regimes:

  1. Form Alone: The model is trained only on surface-level text (e.g., mapping a declarative sentence to its question form).
  2. Form & Meaning: In addition to text, the model is trained to map sentences to structured logical forms (e.g., Sleep(ιx.Newt(x))).

The hypothesis is that because logical meanings are inherently hierarchical, learning to map linear strings to these structures provides a "scaffold" that encourages the model to adopt hierarchical syntactic rules for the text itself.

Key Results: Faster Learning and Better Generalization

The inclusion of semantic signals led to a dramatic shift in how models generalized to unseen, ambiguous examples. In the Seq2Seq setup, models trained on Form Alone generalized linearly 95% of the time. However, those trained on Form & Meaning shifted their preference toward hierarchical generalization.

Training dynamics showing consistency with hierarchical generalization over time for both Language Modeling and Seq2Seq tasks.

Beyond the final accuracy, the semantic signal significantly accelerated the learning process. As shown in the training curves, models with access to meaning reached high levels of hierarchical consistency much faster and with significantly less variance across different random initializations than those trained on form alone. This suggests that semantics makes the "grokking" of hierarchical rules a more reliable and efficient process.

The Role of Agreement and Structural Cues

Ablation studies revealed that the benefit of semantics isn't just about having more data. A critical factor is the presence of subject-auxiliary agreement (e.g., "The newt does" vs "The newts do"). This agreement acts as a "tag" that helps the model identify which auxiliary is the "main" one associated with the subject, rather than one nested within a relative clause. When agreement cues were removed, the models' ability to learn hierarchical rules was severely compromised.

Conclusion and Future Directions

This research demonstrates that semantic information provides a powerful inductive bias for learning hierarchical syntax. By grounding language in meaning, Transformers can overcome the preference for linear shortcuts that plagues text-only training.

Future work aims to test these findings with more realistic, "grounded" semantic signals—such as images or video—and to examine whether these cues are present in the naturalistic data children encounter during development. Understanding these mechanisms may lead to more robust and human-like language models that truly understand the structural foundations of grammar.

Summary updated

BibTeX
@inproceedings{yedetore2024semantic,
  author = {Aditya Yedetore and Najoung Kim},
  title = {{Semantic Training Signals Promote Hierarchical Syntactic Generalization in Transformers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.235},
}
Figure