Introduction
In the era of Contextual Word Representations (CWRs) like ELMo and BERT, models learn a great deal about language structure implicitly by predicting words in context. However, a persistent question in NLP research is whether explicitly injecting "inductive biases"—linguistic rules or structures—can further enhance performance. This paper investigates whether teaching models about shallow syntax (basic phrase groupings or "chunking") improves their ability to handle downstream tasks.
The mSynC Architecture
To test the utility of shallow syntax, the authors developed mSynC (Contextualization via Shallow Syntactic Context), a novel pretraining architecture. Unlike standard language models that predict the next word based solely on the preceding word sequence, mSynC conditions its predictions on the last completed syntactic chunk.
The architecture employs two encoders: a sequential encoder for word-level context and a "Shallow-Syntactic Transformer" to process the history of chunks. This setup forces the model to explicitly track syntactic boundaries during pretraining. To manage training costs, the authors used a staged approach, initializing the sequential encoder with pretrained ELMo weights.
Experimental Results
The authors evaluated mSynC against standard ELMo baselines across four major benchmarks: Coarse-grained NER (CoNLL 2003), Fine-grained NER (OntoNotes 5.0), Constituency Parsing (Penn Treebank), and Sentiment Analysis (SST-5).
The results, summarized in the table above, reveal a surprising trend: the gains from explicit shallow syntax are negligible. For instance, on Coarse NER, mSynC achieved an F1 of 91.5, identical to the ELMo baseline. Similarly, sentiment analysis accuracy remained unchanged at 53.0%. While there was a marginal improvement in constituency parsing compared to original ELMo reports, the authors' own reimplementation of ELMo performed just as well as the syntax-aware mSynC.
Discussion: The Redundancy of Explicit Syntax
The core insight of this study is that explicit shallow syntax is largely redundant for modern deep models. Probing tasks confirmed that mSynC did learn chunking better than standard ELMo (96.9 vs 92.2 F1), yet this superior structural knowledge did not translate into better performance on semantic or syntactic downstream tasks.
This suggests that ELMo-style pretraining already implicitly captures the necessary structural information. Forcing the model to focus on shallow chunks may even be counterproductive; probing results showed mSynC performed slightly worse on several other linguistic tasks, indicating that the explicit syntactic objective might distract the model from learning other useful features. Ultimately, while shallow syntax is computationally cheaper than full parsing, it appears to be "too shallow" to provide a meaningful boost over what deep models learn from raw text alone.
Summary updated