Abstract
Recent analyses suggest that encoders pretrained for language modeling capture certain morpho-syntactic structure. However, probing frameworks for word vectors still do not report results on standard setups such as constituent and dependency parsing. This paper addresses this problem and does full parsing (on English) relying only on pretraining architectures -- and no decoding. We first cast constituent and dependency parsing as sequence tagging. We then use a single feed-forward layer to directly map word vectors to labels that encode a linearized tree. This is used to: (i) see how far we can reach on syntax modelling with just pretrained encoders, and (ii) shed some light about the syntax-sensitivity of different word vectors (by freezing the weights of the pretraining network during training). For evaluation, we use bracketing F1-score and LAS, and analyze in-depth differences across representations for span lengths and dependency displacements. The overall results surpass existing sequence tagging parsers on the PTB (93.5%) and end-to-end EN-EWT UD (78.8%).
Introduction
In Natural Language Processing (NLP), parsing—the task of determining the grammatical structure of a sentence—has traditionally required complex, specialized machinery. Whether identifying phrase groupings (constituent parsing) or head-dependent relationships (dependency parsing), state-of-the-art systems often rely on intricate transition-based or graph-based algorithms and hand-crafted features.
While recent research suggests that modern pretrained encoders like BERT and ELMo capture syntactic information, most studies have focused on simplified "probing" tasks. This paper investigates a more ambitious question: Can full, professional-grade parsing be achieved using only a pretrained encoder and a minimal linear layer, effectively offloading the "intelligence" of the parser to the pretraining phase?
Methodology
The researchers adopt a minimalist approach by casting parsing as a sequence tagging problem. This involves two key steps:
- Linearization: Complex tree structures are flattened into a sequence of labels, one for every word in the sentence. For constituent trees, this is achieved using a 3-tuple encoding tree depth and phrase types.
- Minimalist Architecture: Instead of employing complex decoders like BiLSTMs or Pointer Networks, the model uses a single feed-forward (FF) layer. This layer maps the word vectors directly from the encoder to the predicted tags.
The study evaluates two scenarios: a frozen setup, where encoder weights are locked to probe inherent knowledge, and a fine-tuned setup, to observe the maximum potential of the architecture.
Results and Analysis
The results demonstrate that pretraining effectively acts as a parser. A fine-tuned BERT model with just a linear layer achieved a 93.5% F1 score on the Penn Treebank (PTB), surpassing previous sequence tagging parsers and even strong traditional models. For dependency parsing on the EN-EWT dataset, the model reached a 78.8% Labeled Attachment Score (LAS).
Analysis of span lengths (Figure 4) reveals that while performance naturally decreases as spans become longer and more complex, contextualized models like BERT and ELMo maintain significantly higher accuracy than static embeddings (e.g., GloVe, word2vec). This suggests that the structural knowledge required for parsing is an emergent property of contextualized language modeling. Notably, even a frozen BERT model achieved a 78.2% F1 on PTB, proving that hierarchical structure is deeply embedded in the representation before any task-specific training.
Discussion and Limitations
The study highlights that contextualized representations are far superior to static ones for structural tasks, likely because the objective of predicting words based on neighbors aligns with learning phrase clusters. However, some limitations remain:
- Language Scope: The analysis is currently limited to English; the effectiveness of this linear approach on morphologically rich or free-word-order languages is yet to be determined.
- Lossy Linearization: Sequence tagging is a "lossy" representation. Predicting tags word-by-word can occasionally result in "illegal" trees (e.g., cycles), requiring post-processing rules for correction.
Future work may involve "control tasks" to further isolate how much syntactic intent is truly inside the encoder versus the pattern-matching capability of the linear probe.
Summary updated