Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models

Taeuk Kim, Bowen Li, Sang-goo Lee

Abstract: Taeuk Kim, Bowen Li, Sang-goo Lee. Findings of the Association for Computational Linguistics: EMNLP 2021.

  1. This paper explores how to extract inherent structural knowledge from large pre-trained models without explicit parser training.

Introduction

Constituency parsing—the task of breaking a sentence into a hierarchical tree of phrases—usually requires massive amounts of human-annotated data, which is unavailable for most languages. While unsupervised parsing learns from raw text, these models are often language-specific and difficult to tune. A newer alternative, Constituency Parse Extraction from PLMs (CPE-PLM), leverages the latent knowledge inside models like BERT. However, previous CPE-PLM methods were often greedy and lagged behind dedicated unsupervised parsers. This paper introduces a novel chart-based method and a "top-K ensemble" technique to significantly narrow this gap.

Methodology

The authors introduce two major improvements to the CPE-PLM paradigm:

  1. Chart-based Extraction: Instead of a greedy top-down approach that makes irreversible local decisions, the authors use a chart-parsing algorithm (inspired by CKY). This considers all possible sub-phrases (spans) and uses dynamic programming to find the globally most probable tree structure. They propose two scoring functions: Pair Score ($s_p$), measuring similarity between word attention distributions, and Characteristic Score ($s_c$), comparing word attention to the span's average.
  2. Top-K Ensemble: Syntactic information is often distributed across multiple attention heads. The authors select the top $K$ (e.g., 20) attention heads based on a small validation set and average their outputs to produce a more robust final tree.
Performance comparison of different extraction methods across various PLMs, showing the significant boost from chart-based methods (CP, CC) and Top-K ensembles.

Key Results and Insights

The proposed method achieved a new state-of-the-art F1 score of 46.4 on the English Penn Treebank. In many languages, such as Korean and Swedish, this zero-shot extraction method outperformed traditional unsupervised Probabilistic Context-Free Grammars (PCFGs).

Universal Syntactic Heads

A striking finding is that attention heads optimized for English syntax work remarkably well for other languages. This cross-lingual transfer implies that multilingual PLMs learn a "universal" internal representation of phrase relationships. Furthermore, multilingual models (like XLM-R) often outperformed monolingual models on their native languages, suggesting that learning multiple languages simultaneously forces the model to develop a more robust understanding of syntax.

Visualization of attention heads in XLM-R, demonstrating that specific heads consistently capture syntactic information across diverse languages.

Limitations and Future Work

While powerful, the method is not "purely" unsupervised as it requires a small validation set to select attention heads. Computationally, chart parsing is $O(n^3)$, which is slower than simpler greedy methods. Additionally, analysis showed the model is excellent at identifying Noun Phrases (NPs) but struggles with complex, longer Verb Phrases (VPs), highlighting a need for better modeling of long-range dependencies in future research.

Conclusion

By shifting from greedy to chart-based extraction and utilizing ensembles of attention heads, this work demonstrates that pre-trained language models contain rich, extractable syntactic knowledge that can be applied across diverse languages with minimal supervision.

Summary updated

BibTeX
@inproceedings{kim2021multilingual,
  author = {Taeuk Kim and Bowen Li and Sang-goo Lee},
  title = {{Multilingual Chart-based Constituency Parse Extraction from Pre-trained Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=XEynSc-1NEw},
}
Figure