An Imitation Learning Approach to Unsupervised Parsing

Bowen Li, Lili Mou, Frank Keller

Abstract

Recently, there has been an increasing interest in unsupervised parsers that optimize semantically oriented objectives, typically using reinforcement learning. Unfortunately, the learned trees often do not match actual syntax trees well. Shen et al. (2018) propose a structured attention mechanism for language modeling (PRPN), which induces better syntactic structures but relies on ad hoc heuristics. Also, their model lacks interpretability as it is not grounded in parsing actions. In our work, we propose an imitation learning approach to unsupervised parsing, where we transfer the syntactic knowledge induced by the PRPN to a Tree-LSTM model with discrete parsing actions. Its policy is then refined by Gumbel-Softmax training towards a semantically oriented objective. We evaluate our approach on the All Natural Language Inference dataset and show that it achieves a new state of the art in terms of parsing $F$-score, outperforming our base models, including the PRPN.

Introduction

In Natural Language Processing (NLP), understanding the hierarchical structure of sentences—the way words group into nested constituents like noun or verb phrases—is fundamental for tasks such as machine translation and summarization. Traditionally, parsers are trained on "treebanks" manually annotated by linguists. However, these resources are expensive to produce and unavailable for the vast majority of languages.

Unsupervised parsing seeks to bridge this gap by learning syntactic structures directly from raw text. While Reinforcement Learning (RL) has been applied to optimize models for secondary tasks, these approaches often suffer from instability and produce trees that diverge significantly from human grammar. The Parsing-Reading-Predict Network (PRPN) offers a promising alternative by inducing syntactic structure through structured attention, but it relies on ad-hoc heuristics to generate discrete trees, limiting its interpretability and end-to-end trainability.

Methodology: A Three-Stage Framework

We address these limitations through a novel "teacher-student" architecture that combines the strengths of continuous language models and discrete parsers.

1. Knowledge Transfer and Imitation Learning

The process begins with the PRPN acting as a "teacher." The PRPN learns a continuous "syntactic distance" metric, representing the relative height at which two words are joined. A "student" model—a Tree-LSTM—is then introduced to learn discrete parsing actions (e.g., deciding which adjacent words to merge). By using the syntactic distances provided by the PRPN, we derive a target sequence of operations, allowing the Tree-LSTM to mimic the teacher's implicit knowledge.

Overview of the knowledge transfer process from the PRPN (teacher) to the Tree-LSTM (student).

2. Policy Refinement

After the imitation phase, the student model is fine-tuned on a semantic objective: Natural Language Inference (NLI). To handle the discrete nature of parsing decisions during backpropagation, we employ the Straight-Through Gumbel-Softmax trick. This allows the model to refine its parsing policy based on semantic feedback while maintaining a discrete decision-making process.

Results and Discussion

We evaluated our model on the AllNLI dataset, comparing induced trees against gold-standard trees from the Stanford Parser. Our "Imitation + Refine" model achieved a new state-of-the-art parsing F-score of 53.7, significantly outperforming the PRPN teacher (51.6) and a standalone Tree-LSTM (21.9).

Qualitative Analysis and Robustness

A visual comparison of the trees induced by our model versus the PRPN baseline reveals that our approach produces structures that more closely align with human linguistic intuition, particularly in identifying Verb Phrases (VP) and Sentences (S).

Comparison of induced trees between the PRPN baseline (a) and our proposed imitation learning model (b).

Furthermore, our approach demonstrated significant robustness. In experiments where punctuation was removed—a common "cheat" for unsupervised models—our imitation learning framework outperformed the PRPN by a margin of 7.3 F-score points. This suggests that the model is learning genuine syntactic patterns rather than relying on surface-level cues.

Conclusion

This work demonstrates that imitation learning can successfully bridge the gap between continuous, heuristic-based models and discrete, interpretable parsers. By initializing a discrete parser with the knowledge of a language model and refining it with semantic feedback, we achieve superior syntactic quality and stability. Future research will explore ensembling techniques to address current weaknesses in Noun Phrase (NP) identification.

Summary updated

Abstract

Recently, there has been an increasing interest in unsupervised parsers that optimize semantically oriented objectives, typically using reinforcement learning. Unfortunately, the learned trees often do not match actual syntax trees well. Shen et al. (2018) propose a structured attention mechanism for language modeling (PRPN), which induces better syntactic structures but relies on ad hoc heuristics. Also, their model lacks interpretability as it is not grounded in parsing actions. In our work, we propose an imitation learning approach to unsupervised parsing, where we transfer the syntactic knowledge induced by the PRPN to a Tree-LSTM model with discrete parsing actions. Its policy is then refined by Gumbel-Softmax training towards a semantically oriented objective. We evaluate our approach on the All Natural Language Inference dataset and show that it achieves a new state of the art in terms of parsing $F$-score, outperforming our base models, including the PRPN.
BibTeX
@inproceedings{li2019imitation,
  author = {Bowen Li and Lili Mou and Frank Keller},
  title = {{An Imitation Learning Approach to Unsupervised Parsing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=fwgbJA50y5x},
}
Figure