Revisiting Graph Meaning Representations through Decoupling Contextual Representation Learning and Structural Information Propagation

Li Zhou, Wenyu Chen, Dingyi Zeng, Hong Qu, Daniel Hershcovich

Abstract

In the field of natural language understanding, the intersection of neural models and graph meaning representations (GMRs) remains a compelling area of research. Despite the growing interest, a critical gap persists in understanding the exact influence of GMRs, particularly concerning relation extraction tasks. Addressing this, we introduce DAGNN-plus, a simple and parameter-efficient neural architecture designed to decouple contextual representation learning from structural information propagation. Coupled with various sequence encoders and GMRs, this architecture provides a foundation for systematic experimentation on two English and two Chinese datasets. Our empirical analysis utilizes four different graph formalisms and nine parsers. The results yield a nuanced understanding of GMRs, showing improvements in three out of the four datasets, particularly favoring English over Chinese due to highly accurate parsers. Interestingly, GMRs appear less effective in literary-domain datasets compared to general-domain datasets. These findings lay the groundwork for better-informed design of GMRs and parsers to improve relation classification, which is expected to tangibly impact the future trajectory of natural language understanding research.

Introduction: The Challenge of Belief Bias

Relation Classification (RC)—identifying the relationship between two entities in a sentence—is a cornerstone of information extraction. While Pre-trained Language Models (PLMs) like BERT are the current state-of-the-art, they often suffer from what researchers call "belief bias." Because PLMs are trained on massive corpora, they frequently rely on statistical word associations rather than the actual syntactic structure of a specific sentence.

For example, a PLM might see the word "wife" and predict a "spouse" relationship, even if the sentence structure clearly indicates a "parent-child" relationship (e.g., "Jong-Un was born to the leader's third wife Ko Yong-Hi"). Explicit structural information from Graph Meaning Representations (GMRs), such as dependency trees, can act as a structural anchor to override these statistical biases by highlighting the shortest path between entities.

Figure 1: Examples of PLM belief bias versus GMR-based method predictions in English and Chinese.

Methodology: The DAGNN-plus Architecture

Previous attempts to integrate GMRs often used complex Graph Neural Networks (GNNs) that intertwined representation learning with graph propagation. This complexity made it difficult to discern whether improvements stemmed from the structural information or simply from the addition of model parameters.

To address this, we propose DAGNN-plus, based on the Decoupled Aggregation Graph Neural Network. The core philosophy is the strict decoupling of two processes:

  1. Contextual Learning: A standard PLM (such as BERT or RoBERTa) encodes raw text into hidden representations.
  2. Structural Propagation: A parameter-efficient graph encoder propagates these representations across the edges of a GMR.
Figure 2: The DAGNN-plus architecture, illustrating the decoupling of contextual learning and structural propagation.

This architecture is highly efficient; while a standard GCN adds $L \times d^2$ parameters, DAGNN-plus adds only $d$ parameters—essentially a single vector to learn the "receptiveness" of each layer. Furthermore, we implemented a subword integration strategy to map word-level dependency edges to the subword units used by modern PLMs.

Experimental Setup and Results

We evaluated DAGNN-plus across four benchmarks: Re-TACRED and SemEval 2010 Task 8 for English, and FinRE (financial news) and SanWen (Chinese literature) for Chinese. Our study tested both syntactic graphs (grammatical relations) and semantic graphs (meaning-based relations).

Key Findings

  • Consistent Improvements: DAGNN-plus outperformed the Improved RC (IRC) baseline on three out of four datasets. On Re-TACRED, adding Universal Dependencies (UD) or Semantic Dependencies (DM) improved F1 scores by 0.5 to 1.0 points. In the Chinese FinRE dataset, we observed a significant boost of 1.17 F1 points.
  • The "Literary Wall": GMRs failed to provide benefits on the SanWen dataset. This highlights a "domain gap": the parsers used to generate the GMRs were trained on news text. When faced with the complex imagery and non-standard syntax of Chinese literature, the parsers produced noisy graphs that misled the model.
  • Parser Quality Correlation: Our results show a direct correlation between a parser's intrinsic accuracy and its extrinsic value for the RC task. High-quality parsers are essential for GMRs to be effective.

Conclusion

By decoupling contextual learning from structural propagation, we have demonstrated that explicit structural information consistently improves relation classification in general domains. However, the effectiveness of GMRs is heavily dependent on parser quality, particularly when moving into specialized domains like literature. Future work should focus on developing domain-specific parsers and standardizing GMR frameworks for non-English languages to further unlock the potential of structural information in natural language understanding.

Summary updated

Abstract

In the field of natural language understanding, the intersection of neural models and graph meaning representations (GMRs) remains a compelling area of research. Despite the growing interest, a critical gap persists in understanding the exact influence of GMRs, particularly concerning relation extraction tasks. Addressing this, we introduce DAGNN-plus, a simple and parameter-efficient neural architecture designed to decouple contextual representation learning from structural information propagation. Coupled with various sequence encoders and GMRs, this architecture provides a foundation for systematic experimentation on two English and two Chinese datasets. Our empirical analysis utilizes four different graph formalisms and nine parsers. The results yield a nuanced understanding of GMRs, showing improvements in three out of the four datasets, particularly favoring English over Chinese due to highly accurate parsers. Interestingly, GMRs appear less effective in literary-domain datasets compared to general-domain datasets. These findings lay the groundwork for better-informed design of GMRs and parsers to improve relation classification, which is expected to tangibly impact the future trajectory of natural language understanding research.
BibTeX
@misc{zhou2023revisiting,
  author = {Li Zhou and Wenyu Chen and Dingyi Zeng and Hong Qu and Daniel Hershcovich},
  title = {{Revisiting Graph Meaning Representations through Decoupling Contextual Representation Learning and Structural Information Propagation}},
  year = {2023},
  eprint = {2310.09772},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.09772},
}
Figure