Abstract: Taeuk Kim, Bowen Li, Sang-goo Lee. Findings of the Association for Computational Linguistics: EMNLP 2021.
- This paper explores how to extract inherent structural knowledge from large pre-trained models without explicit parser training.
Introduction
Constituency parsing—the task of breaking a sentence into a hierarchical tree of phrases—usually requires massive amounts of human-annotated data, which is unavailable for most languages. While unsupervised parsing learns from raw text, these models are often language-specific and difficult to tune. A newer alternative, Constituency Parse Extraction from PLMs (CPE-PLM), leverages the latent knowledge inside models like BERT. However, previous CPE-PLM methods were often greedy and lagged behind dedicated unsupervised parsers. This paper introduces a novel chart-based method and a "top-K ensemble" technique to significantly narrow this gap.
Methodology
The authors introduce two major improvements to the CPE-PLM paradigm:
- Chart-based Extraction: Instead of a greedy top-down approach that makes irreversible local decisions, the authors use a chart-parsing algorithm (inspired by CKY). This considers all possible sub-phrases (spans) and uses dynamic programming to find the globally most probable tree structure. They propose two scoring functions: Pair Score ($s_p$), measuring similarity between word attention distributions, and Characteristic Score ($s_c$), comparing word attention to the span's average.
- Top-K Ensemble: Syntactic information is often distributed across multiple attention heads. The authors select the top $K$ (e.g., 20) attention heads based on a small validation set and average their outputs to produce a more robust final tree.
Key Results and Insights
The proposed method achieved a new state-of-the-art F1 score of 46.4 on the English Penn Treebank. In many languages, such as Korean and Swedish, this zero-shot extraction method outperformed traditional unsupervised Probabilistic Context-Free Grammars (PCFGs).
Universal Syntactic Heads
A striking finding is that attention heads optimized for English syntax work remarkably well for other languages. This cross-lingual transfer implies that multilingual PLMs learn a "universal" internal representation of phrase relationships. Furthermore, multilingual models (like XLM-R) often outperformed monolingual models on their native languages, suggesting that learning multiple languages simultaneously forces the model to develop a more robust understanding of syntax.
Limitations and Future Work
While powerful, the method is not "purely" unsupervised as it requires a small validation set to select attention heads. Computationally, chart parsing is $O(n^3)$, which is slower than simpler greedy methods. Additionally, analysis showed the model is excellent at identifying Noun Phrases (NPs) but struggles with complex, longer Verb Phrases (VPs), highlighting a need for better modeling of long-range dependencies in future research.
Conclusion
By shifting from greedy to chart-based extraction and utilizing ensembles of attention heads, this work demonstrates that pre-trained language models contain rich, extractable syntactic knowledge that can be applied across diverse languages with minimal supervision.
Summary updated