Improving Text Understanding via Deep Syntax-Semantics Communication

Hao Fei, Yafeng Ren, Donghong Ji

Abstract

This paper introduces a deep neural communication model designed to bridge the gap between syntactic tree encoders and sequential semantic encoders. Unlike traditional shallow integration methods that simply concatenate outputs, this approach facilitates a multi-stage, interactive exchange of information. The model achieves state-of-the-art results on syntax-dependent tasks, including Semantic Role Labeling (SRL) and Natural Language Inference (NLI), by allowing these two distinct perspectives to mutually refine their representations during the learning process.

Introduction

Modeling natural language typically involves two complementary approaches: sequential modeling, which captures local context and word order (e.g., LSTMs or Transformers), and syntactic modeling, which captures the hierarchical structure of sentences (e.g., dependency trees). While both are essential for deep understanding, they are often treated as independent components in existing NLP architectures.

Standard "shallow integration" techniques—such as concatenating final hidden states—fail to capture the dynamic interplay between structure and sequence. This can lead to blind spots where a sequential model misses long-distance dependencies or a tree model overlooks nuanced local context. The authors argue that without a mechanism for communication during the encoding process, these models cannot effectively correct each other's errors.

Deep Communication Architecture

The proposed Deep Syntax-Semantics Communication model addresses these limitations through two primary sub-models: a sequential encoder (BiLSTM) and a tree encoder (TreeLSTM or GCN). The core innovation lies in two communication mechanisms:

  1. Local Interaction: At each word position, the encoders exchange information using an attentive operation. As shown in Figure 5, the sequential state $h_i^{seq}$ and tree state $h_j^{tree}$ are updated by attending to their counterparts in the previous time step, allowing for a cross-perspective feature fusion.
Local interaction between sequential and tree encoders.
  1. Global Propagation: The model iterates over $T$ recurrent steps. In each step, a context gate regulates the flow of information from the previous iteration, enabling global propagation that effectively mitigates the "long-range dependency" problem where models lose information from the beginning of a sentence.

Experimental Results

The model was evaluated on several syntax-heavy benchmarks, including Event Factuality Prediction (EFP), Drug-Drug Interaction (DDI) relation classification, Semantic Role Labeling (SRL) on CoNLL08, and Natural Language Inference (NLI) across QNLI, SICK, and RTE. Dependency trees were generated using a state-of-the-art BiAffine parser.

The proposed model consistently outperformed strong baselines, including Transformers and multi-task learning models. For instance, on the QNLI dataset, the model achieved 88.2% accuracy, a significant improvement over the standalone BiLSTM (78.6%) and even the ensemble approach (84.6%). In SRL tasks, the model reached an F1 score of 85.7, outperforming the S-LSTM baseline of 83.0.

Robustness and Analysis

A key finding of the research is the model's robustness to sentence length. Standard models usually experience a performance drop as sentences grow longer. However, as illustrated in Figure 6, the proposed model's global propagation mechanism acts as a buffer, maintaining high performance (measured by Pearson's $r$) even when sentence length exceeds 30 words.

Model performance (Pearson's r) vs. sentence length.

Further analysis revealed that syntax and semantics are highly interdependent. Case studies showed that after several communication steps (typically $T=5$), the attention weights of the two models begin to align, suggesting they reach a consensus on which words are most critical for understanding the sentence.

Conclusion

The deep communication framework demonstrates that fostering mutual learning between syntactic and semantic representations leads to significant gains in accuracy and robustness. While the model introduces some computational overhead due to its iterative nature, the performance benefits on complex, syntax-dependent tasks are substantial. Future work may involve integrating this architecture directly into the internal layers of large-scale Transformer-based models to further enhance their structural awareness.

Summary updated

BibTeX
@inproceedings{fei2020improving,
  author = {Hao Fei and Yafeng Ren and Donghong Ji},
  title = {{Improving Text Understanding via Deep Syntax-Semantics Communication}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=LICXaMUv3jR},
}
Figure