Hao Fei, Yafeng Ren, Donghong Ji. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.
Introduction
Large-scale Transformer models like BERT have revolutionized NLP by capturing complex linguistic patterns through massive pre-training. However, their understanding of syntax—the structural rules of language—remains largely implicit. Probing studies suggest that while these models capture some grammar, they often struggle with tasks requiring deep linguistic nuance, such as semantic role labeling (SRL) or complex relation classification. This paper introduces a novel approach to "retrofit" Transformers with explicit syntactic knowledge by injecting structural learning into specific middle layers.
The Challenge: Implicit Syntax and Layer Interference
Existing attempts to integrate syntax into Transformers, often called "Tree Transformers," face two significant hurdles:
- Layer Interference: Most models apply structural constraints across all layers. However, research indicates that syntax is best represented in the middle layers of deep networks, while upper layers should focus on high-level semantics. Forcing syntax into the final layers can degrade the model's ultimate semantic representations.
- Structural Monoculture: Most models focus on either constituency (phrasal grouping) or dependency (relational links), but rarely both. Different downstream tasks require different syntactic "flavors."
Methodology: Injecting Explicit Structure
The authors propose a Structure-aware Transformer that utilizes a specialized Structure Learning Module integrated via a multi-task learning objective.
Syntactic Distance
To represent tree structures efficiently, the model employs "syntactic distance." Instead of complex tree-traversal algorithms, it uses a sequence of scalar values where the relative distances between adjacent words reflect their height in a parse tree. This mathematically unified approach allows the model to simultaneously represent both phrases and dependencies.
Middle-Layer Integration and the Structure Learning Module
Based on empirical probing, the authors inject structural learning specifically at Layer 6 (in a 12-layer BERT-base architecture). This ensures that lower layers learn basic features, middle layers handle syntax, and top layers remain optimized for semantic tasks. The Structure Learning Module uses a phrasal attention mechanism to group words into coherent units based on latent context.
During pre-training, the model optimizes a joint objective: standard Masked Language Modeling (MLM) and a Structure Induction loss. During fine-tuning, the structure module remains active, allowing the model to adapt its internal syntax to the specific needs of a task (e.g., favoring noun phrases for sentiment analysis or relational links for SRL).
Key Results and Insights
- Language Modeling Performance: The retrofitted model achieved a perplexity of 37.0 (unsupervised) and 29.2 (supervised), a massive improvement over the vanilla Transformer baseline of 78.6.
- Downstream Task Gains: Across a suite of eight tasks, including NER, Sentiment Analysis (SST), and SRL, the proposed model raised the average score from 80.16 to 83.68.
- The "Sweet Spot" for Syntax: Probing confirmed that Layer 6 is the peak for syntactic information. Injecting structure here prevents the "semantic wash-out" observed when constraints are applied to the final layers.
- Unsupervised Grammar Induction: In unsupervised constituency parsing, the model achieved a 60.3 F1 score, outperforming several specialized recursive neural network models designed specifically for grammar induction.
Conclusion
By retrofitting Transformers with a middle-layer structure module and utilizing the unified representation of syntactic distance, this work demonstrates that explicit syntax significantly enhances both language modeling and downstream semantic performance. The model's ability to induce latent structures—even without "gold" trees—suggests a robust path forward for building more linguistically-informed AI.
Summary updated