Cooperative Learning of Disjoint Syntax and Semantics

Serhii Havrylov, Germán Kruszewski, Armand Joulin

Abstract

Latent tree models aim to induce syntactic structure while learning semantic representations. However, these models often suffer from a coadaptation problem where the continuous composition function outpaces the discrete parser, leading to suboptimal syntactic discovery. This paper introduces a decoupled architecture using Proximal Policy Optimization (PPO) to synchronize the learning of syntax and semantics. Our approach achieves near-perfect accuracy (99.2%) on the challenging ListOps benchmark, a significant improvement over prior state-of-the-art models.

Introduction

In linguistics, the principle of compositionality suggests that the meaning of a complex expression is determined by the meanings of its constituent parts and the rules used to combine them. Latent tree models attempt to operationalize this by jointly learning a parser (syntax) and a composition function (semantics). A persistent challenge in this field is the "coadaptation problem": the continuous composition function typically learns much faster than the discrete parser. Consequently, the parser often fails to discover meaningful grammar, instead adopting trivial structures that satisfy the immediate needs of the composition function. This failure is particularly evident in logical tasks like ListOps, where correct hierarchical processing is mandatory.

Methodology: Synchronized Learning

The proposed model enforces a strict separation between the Parser and the Composition Function. The Parser is a discrete agent trained via Reinforcement Learning (REINFORCE) to build tree structures. The Composition Function is a Tree-LSTM that computes meanings based on these structures, trained via standard gradient descent.

To address the optimization imbalance, the authors introduce three key innovations:

  1. Proximal Policy Optimization (PPO): PPO is used to constrain the parser's updates, preventing drastic strategy shifts that would destabilize the composition function. This acts as a synchronization mechanism between the two components.
  2. Self-Critical Training (SCT): To reduce the variance of RL gradients without introducing the bias common in differentiable relaxations (like Gumbel-Softmax), the model employs a self-critical baseline.
  3. Untied Parameters: The parser and composer do not share weights, allowing each to specialize in its respective domain.
Ablation study showing the impact of PPO and Self-critical training on model performance.

As shown in the ablation study above, the combination of PPO and Self-critical training is vital. Without these components, the model fails to reach high accuracy, highlighting the importance of stabilized, unbiased optimization in latent tree induction.

Experimental Results

The model was evaluated on ListOps (a diagnostic logic task), the Stanford Sentiment Treebank (SST), and Natural Language Inference (SNLI/MultiNLI) datasets.

ListOps Breakthrough

ListOps serves as a rigorous test for a model's ability to learn context-free grammars. Previous latent tree models, such as the Gumbel Tree-LSTM, have struggled significantly on this task. Our model achieves a breakthrough accuracy of 99.2%.

Comparison of the proposed model against standard LSTM and previous latent tree models on the ListOps benchmark.

The results in the table above demonstrate that the proposed method far exceeds the performance of both standard LSTMs and previous reinforcement learning-based tree models (RL-SPINN).

Natural Language and Generalization

On natural language tasks like SST and SNLI, the model performs competitively with existing state-of-the-art latent tree models. While natural language often allows for "shortcut" strategies due to its inherent redundancy, the model still manages to find useful structures. Notably, the model exhibits exceptional extrapolation capabilities; when trained on sequences of ~100 tokens, it maintains high accuracy on sequences exceeding 1000 tokens, indicating it has learned the underlying grammatical rules rather than just surface patterns.

Conclusion

This work demonstrates that the failure of previous latent tree models was largely an optimization issue rather than an architectural one. By using PPO to synchronize the learning of syntax and semantics and employing unbiased gradient estimators, we can successfully induce complex hierarchical structures. Future research will explore applying these techniques to large-scale language modeling and more sophisticated RL reward structures.

Summary updated

BibTeX
@inproceedings{havrylov2019cooperative,
  author = {Serhii Havrylov and Germán Kruszewski and Armand Joulin},
  title = {{Cooperative Learning of Disjoint Syntax and Semantics}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=H1-1QmZuZr},
}
Figure