Evaluating distillation methods for data-efficient syntax learning

Takateru Yamakoshi, Thomas L. Griffiths, R. Thomas McCoy, Robert D. HawkinsEMNLP 2025 · OpenReview

Abstract

Data-efficient training requires strong inductive biases. To the extent that transformer attention matrices encode syntactic relationships, we would predict that knowledge distillation (KD) targeting attention should selectively accelerate syntax acquisition relative to conventional logit-based KD. To test this hypothesis, we train GPT-2 student models on datasets ranging from 10K to 5M sentences using both distillation methods, evaluating them on both syntactic benchmarks and perplexity. Surprisingly, while logit-based KD dramatically improves data-efficiency, attention-based KD provides minimal benefit even for syntactic tasks. This suggests that output distributions provide sufficient supervisory signal for syntax acquisition, indicating that syntactic knowledge may be distributed throughout the network rather than localized in attention patterns.

Introduction

Modern Large Language Models (LLMs) require billions of tokens to achieve fluency, far exceeding the data exposure of a human child. To bridge this gap, researchers investigate inductive biases—predispositions that help models learn structural rules from limited data. A prominent hypothesis in interpretability research suggests that the Transformer's attention mechanism is the primary locus of syntactic processing. If specific attention heads are indeed "syntax experts," forcing a student model to mimic a teacher's attention patterns through Knowledge Distillation (KD) should theoretically accelerate grammar acquisition.

Methodology

We conducted a controlled experiment using GPT-2 Small for both the pre-trained teacher and the student trained from scratch. By maintaining identical architectures, we isolated the impact of the distillation objective. We compared three primary training conditions across dataset sizes ranging from 10,000 to 5 million sentences:

  1. Baseline: Standard training without a teacher.
  2. Logit-based KD: The student matches the teacher's final word-prediction probabilities (KL divergence).
  3. Attention-based KD: The student matches the teacher’s internal attention matrices (Mean Squared Error).
Conceptual diagram of logit-based vs. attention-based knowledge distillation from a teacher to a student model.

Models were trained on the BabyLM dataset, which provides developmentally plausible text scales. Evaluation focused on three syntactic benchmarks—Linzen (subject-verb agreement), BLiMP (broad linguistic phenomena), and Zorro (child-directed speech structures)—alongside standard perplexity metrics.

Results and Discussion

Our results challenge the assumption that attention-based distillation is a superior path to syntactic competence. While logit-based KD enabled the student to reach teacher-level performance with only 500,000 sentences—a 10x improvement in data efficiency over the baseline—attention-based KD yielded only marginal gains.

Performance comparison across benchmarks (Linzen, BLiMP, Zorro, and Perplexity) showing that logit-based KD (orange) significantly outperforms attention-based KD (blue) and the baseline (grey).

Key Findings

  • Logits Carry "Dark Knowledge": The final output distribution contains sufficient signal to teach complex grammar. This suggests that syntactic information is not localized in attention heads but is processed and refined through MLP layers and residual streams before reaching the output.
  • Redundancy and Interference: Combining both KD methods provided no benefit over logit-based KD alone. Furthermore, on the smallest datasets, attention-based distillation occasionally increased perplexity, suggesting that forcing specific internal structures can conflict with the primary objective of next-token prediction.
  • The Attention-Syntax Link: Despite the presence of syntactically-aligned attention heads in pre-trained models, explicitly supervising these patterns does not appear to be an efficient shortcut for learning.

Conclusion

While attention mechanisms are vital for Transformer function, they do not appear to be the most effective target for distilling syntactic knowledge. Logit-based distillation remains the most robust method for improving data efficiency, suggesting that the "dark knowledge" in output distributions is a more complete representation of a model's linguistic competence than its internal attention patterns alone.

Summary updated

Abstract

Data-efficient training requires strong inductive biases. To the extent that transformer attention matrices encode syntactic relationships, we would predict that knowledge distillation (KD) targeting attention should selectively accelerate syntax acquisition relative to conventional logit-based KD. To test this hypothesis, we train GPT-2 student models on datasets ranging from 10K to 5M sentences using both distillation methods, evaluating them on both syntactic benchmarks and perplexity. Surprisingly, while logit-based KD dramatically improves data-efficiency, attention-based KD provides minimal benefit even for syntactic tasks. This suggests that output distributions provide sufficient supervisory signal for syntax acquisition, indicating that syntactic knowledge may be distributed throughout the network rather than localized in attention patterns.
BibTeX
@inproceedings{yamakoshi2025evaluating,
  author = {Takateru Yamakoshi and Thomas L. Griffiths and R. Thomas McCoy and Robert D. Hawkins},
  title = {{Evaluating distillation methods for data-efficient syntax learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=qUrPX3pksi},
}
Figure