Introduction
Syntactic structure—the underlying rules governing how words relate and group—is a cornerstone of natural language understanding. Traditionally, NLP models leverage two primary types of syntax: dependency trees, which capture head-dependent relations between words, and constituency trees, which organize words into hierarchical phrasal units. While both offer valuable signals, integrating them into modern neural architectures presents significant challenges.
Current approaches often force a choice between tree types, missing out on their complementary benefits. Furthermore, tree-based encoders like TreeLSTMs are computationally intensive and difficult to parallelize. Finally, these models rely on external parsers during inference; any errors made by the parser propagate downstream, degrading final performance.
To address these issues, this paper introduces a "Mimic and Conquer" strategy. The authors propose a Heterogeneous Tree Structure Distillation framework that compresses complex syntactic knowledge from multiple tree-based "teacher" models into a lightweight, high-speed sequential "student" model.
The Distillation Framework
The core idea is to use tree-aware models as teachers to guide a simple 3-layer BiLSTM student. The teachers consist of both Graph Convolutional Networks (GCNs) and TreeLSTMs, each specialized in either dependency or constituency structures.
The student model learns through four distinct distillation objectives:
- Output Distillation: Mimicking the teachers' final probability distributions (soft targets).
- Syntactic Feature Distillation: Aligning the student's hidden representations with the teachers' structural features.
- Semantic Learning: Incorporating a masked language modeling task to maintain general semantic understanding.
- Enhanced Structure Injection: Forcing the student to explicitly predict dependency arcs and constituency spans, internalizing the syntactic rules.
To maintain stability during training, the authors employ a "turning gap" strategy, where the student alternates between learning from dependency and constituency teachers every few iterations.
Key Insights and Task Sensitivity
The research demonstrates that dependency and constituency structures provide mutual benefits. Dependency links are particularly effective at capturing long-range relations (crucial for Relation Classification and Semantic Role Labeling), while constituency spans excel at defining argument boundaries (beneficial for Sentiment Analysis).
By distilling both, the student model achieves a "parser-free" advantage. Once trained, it requires only raw text as input. It has effectively internalized the syntactic rules, making it faster and more robust to the errors that typically plague parser-dependent systems.
Performance and Efficiency
The distilled student was evaluated across four key tasks: Relation Classification (Rel), Natural Language Inference (NLI), Sentiment Analysis (SST), and Semantic Role Labeling (SRL).
The results were compelling. The student model consistently outperformed standalone tree models and ensemble methods. For instance, on the SRL task, the student achieved an 85.5 F1 score, significantly higher than the 82.4 F1 average of standalone tree models.
Beyond accuracy, the student offers massive gains in efficiency. It is approximately 3x smaller in parameter count than adversarial training baselines and 4x faster at decoding than ensemble methods.
Furthermore, the model exhibits remarkable robustness. In low-resource scenarios (e.g., using only 20% of training data), the distilled student maintains significantly higher accuracy than standard models, proving that the distilled syntax acts as a powerful inductive bias.
Conclusion
"Mimic and Conquer" provides a practical path for deploying syntax-aware models in real-world applications. By distilling heterogeneous tree structures into a sequential encoder, the authors eliminate the trade-off between structural richness and computational efficiency. While the current work focuses on LSTMs and English, the framework's success suggests that distilling tree structures into Transformer-based architectures is a promising avenue for future research.
Summary based on "Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP" by Hao Fei, Yafeng Ren, and Donghong Ji (EMNLP 2020).
Summary updated