Shallow Syntax in Deep Water

Swabha Swayamdipta, Matthew E. Peters, Brendan Roof, Chris Dyer, Noah A. SmitharXiv 2019 · arXiv

Introduction

In the era of Contextual Word Representations (CWRs) like ELMo and BERT, models learn a great deal about language structure implicitly by predicting words in context. However, a persistent question in NLP research is whether explicitly injecting "inductive biases"—linguistic rules or structures—can further enhance performance. This paper investigates whether teaching models about shallow syntax (basic phrase groupings or "chunking") improves their ability to handle downstream tasks.

The mSynC Architecture

To test the utility of shallow syntax, the authors developed mSynC (Contextualization via Shallow Syntactic Context), a novel pretraining architecture. Unlike standard language models that predict the next word based solely on the preceding word sequence, mSynC conditions its predictions on the last completed syntactic chunk.

The mSynC architecture: a dual-encoder system conditioning predictions on the last completed syntactic chunk.

The architecture employs two encoders: a sequential encoder for word-level context and a "Shallow-Syntactic Transformer" to process the history of chunks. This setup forces the model to explicitly track syntactic boundaries during pretraining. To manage training costs, the authors used a staged approach, initializing the sequential encoder with pretrained ELMo weights.

Experimental Results

The authors evaluated mSynC against standard ELMo baselines across four major benchmarks: Coarse-grained NER (CoNLL 2003), Fine-grained NER (OntoNotes 5.0), Constituency Parsing (Penn Treebank), and Sentiment Analysis (SST-5).

Performance comparison across downstream tasks showing that mSynC provides negligible gains over ELMo baselines.

The results, summarized in the table above, reveal a surprising trend: the gains from explicit shallow syntax are negligible. For instance, on Coarse NER, mSynC achieved an F1 of 91.5, identical to the ELMo baseline. Similarly, sentiment analysis accuracy remained unchanged at 53.0%. While there was a marginal improvement in constituency parsing compared to original ELMo reports, the authors' own reimplementation of ELMo performed just as well as the syntax-aware mSynC.

Discussion: The Redundancy of Explicit Syntax

The core insight of this study is that explicit shallow syntax is largely redundant for modern deep models. Probing tasks confirmed that mSynC did learn chunking better than standard ELMo (96.9 vs 92.2 F1), yet this superior structural knowledge did not translate into better performance on semantic or syntactic downstream tasks.

This suggests that ELMo-style pretraining already implicitly captures the necessary structural information. Forcing the model to focus on shallow chunks may even be counterproductive; probing results showed mSynC performed slightly worse on several other linguistic tasks, indicating that the explicit syntactic objective might distract the model from learning other useful features. Ultimately, while shallow syntax is computationally cheaper than full parsing, it appears to be "too shallow" to provide a meaningful boost over what deep models learn from raw text alone.

Summary updated

Abstract

Shallow syntax provides an approximation of phrase-syntactic structure of sentences; it can be produced with high accuracy, and is computationally cheap to obtain. We investigate the role of shallow syntax-aware representations for NLP tasks using two techniques. First, we enhance the ELMo architecture to allow pretraining on predicted shallow syntactic parses, instead of just raw text, so that contextual embeddings make use of shallow syntactic context. Our second method involves shallow syntactic features obtained automatically on downstream task data. Neither approach leads to a significant gain on any of the four downstream tasks we considered relative to ELMo-only baselines. Further analysis using black-box probes confirms that our shallow-syntax-aware contextual embeddings do not transfer to linguistic tasks any more easily than ELMo's embeddings. We take these findings as evidence that ELMo-style pretraining discovers representations which make additional awareness of shallow syntax redundant.
BibTeX
@misc{swayamdipta2019shallow,
  author = {Swabha Swayamdipta and Matthew E. Peters and Brendan Roof and Chris Dyer and Noah A. Smith},
  title = {{Shallow Syntax in Deep Water}},
  year = {2019},
  eprint = {1908.11047},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.11047},
}
Figure