Abstract: Recent analyses suggest that encoders pretrained for language modeling capture certain morpho-syntactic structure. However, probing frameworks for word vectors still do not report results on standard setups such as constituent and dependency parsing. This paper addresses this problem and does full parsing (on English) relying only on pretraining architectures – and no decoding. We first cast constituent and dependency parsing as sequence tagging. We then use a single feed-forward layer to directly map word vectors to labels that encode a linearized tree. This is used to: (i) see how far we can reach on syntax modelling with just pretrained encoders, and (ii) shed some light about the syntax-sensitivity of different word vectors (by freezing the weights of the pretraining network during training). For evaluation, we use bracketing F1-score and las, and analyze in-depth differences across representations for span lengths and dependency displacements. The overall results surpass existing sequence tagging parsers on the ptb (93.5%) and end-to-end en-ewt ud (78.8%).
Introduction
In the field of Natural Language Processing (NLP), contextualized embeddings like BERT and ELMo have revolutionized performance across tasks. However, quantifying the exact depth of syntactic knowledge within these "black box" models remains a challenge. Traditional probing studies often focus on isolated tasks, such as subject-verb agreement. In contrast, full parsing—reconstructing the complete hierarchical structure of a sentence—usually necessitates complex decoders like BiLSTMs or transition-based systems.
This study investigates whether pretrained encoders can perform full parsing using a minimalist architecture, effectively testing if modern embeddings are rich repositories of syntactic structure rather than mere statistical word clouds.
Methodology
The researchers adopted a minimalist approach to isolate the contribution of the embeddings. By reducing parsing to a sequence labeling problem, they eliminated the need for specialized parsing logic or heavy-duty decoders.
The Linearization Trick
To handle the 2D nature of trees in a 1D sequence tagging framework, the authors utilized a linearization function. Constituent trees were encoded into tags representing tree levels and non-terminal symbols, while dependency trees were similarly flattened into word-to-word relationship labels.
Model Architecture
The model consists of a single feed-forward layer placed directly on top of various pretrained encoders. This setup ensures that any syntactic "intelligence" observed is derived from the embeddings themselves.
Testing was conducted in two modes:
- Frozen (
ff): Encoder weights are locked to probe "out-of-the-box" knowledge. - Fine-tuned (
ff-ft): Encoders are allowed to adapt, testing the maximum potential of the architecture.
Experimental Setup
The models were evaluated on two standard English benchmarks:
- Penn Treebank (PTB): The primary benchmark for constituent (phrase-structure) parsing.
- EN-EWT UD: A dataset from the Universal Dependencies project used for dependency parsing.
Results and Discussion
BERT as a Syntactic Powerhouse
BERT demonstrated exceptional syntactic sensitivity. When fine-tuned, it achieved a 93.5% F1-score on the PTB, surpassing previous state-of-the-art sequence tagging parsers (which typically reached ~91.2%). Even in the frozen state, BERT reached 78.2% F1, vastly outperforming static vectors like word2vec, which hovered around 40%.
Phrases vs. Dependencies
The results indicate that contextual encoders are particularly adept at capturing phrase-structure. The authors hypothesize that language modeling objectives naturally encourage the grouping of words into constituent phrases. As shown in the analysis of span types, BERT and ELMo maintain high performance across various non-terminal categories compared to their static counterparts.
The Importance of Context
Static embeddings (word2vec, FastText, GloVe) performed poorly, often barely exceeding random noise. This confirms that the strength of modern NLP lies in contextual representation. Furthermore, the study found that powerful decoders (like BiLSTMs) often "mask" the inherent weaknesses of embeddings; by removing the decoder, the true raw syntactic ability of the encoders was revealed.
Conclusion
This work demonstrates that pretrained language models like BERT capture a surprising amount of full syntactic structure. By treating parsing as sequence tagging with a simple linear layer, the authors achieved state-of-the-art results without complex decoding machinery. Future work may involve expanding this analysis to morphologically rich languages and incorporating control tasks to further distinguish between true linguistic rule-learning and statistical pattern matching.
Summary updated