Explore Unsupervised Structures in Pretrained Models for Relation Extraction

Xi Yang, Tao Ji, Yuanbin WuEMNLP 2022 · OpenReview

Abstract

Syntactic trees have been widely applied in relation extraction (RE). However, since parsing qualities are not stable on different text domains and a pre-defined grammar may not well fit the target relation schema, the introduction of syntactic structures sometimes fails to improve RE performances consistently. In this work, we study RE models with various unsupervised structures mined from pre-trained language models (e.g., BERT). We show that, similar to syntactic trees, unsupervised structures are quite informative for RE task: they are able to obtain competitive (even the best) performance scores on benchmark RE datasets (ACE05, WebNLG, SciERC). We also conduct detailed analyses on their abilities of adapting new RE domains and influence of noise links in those structures. The results suggest that unsupervised structures are reasonable alternatives of commonly used syntactic structures in relation extraction models.

Introduction

Relation Extraction (RE) is the task of identifying entities and the semantic relationships between them (e.g., identifying that "Einstein" was "born_in" "Germany"). Traditionally, researchers have boosted RE models by feeding them syntactic dependency trees—maps of how words grammatically relate to one another. However, this approach faces three major hurdles: domain mismatch (parsers trained on news fail on scientific text), redundancy (dense trees contain distracting noise), and high computational costs.

This paper explores a "middleman-free" approach: extracting structural information directly from the internal states of Pre-trained Language Models (PLMs) like BERT.

Methods

The researchers propose a "plug-and-play" framework where structural information is mined from a PLM and fed into a Graph Convolutional Network (GCN) to refine word representations.

The proposed framework compares traditional dependency parsing (top) with unsupervised structure mining from PLMs (bottom) to feed into GCNs.

They explored several ways to "mine" these structures:

  • Attention-based: Using internal attention matrices (e.g., raw Attn, Rollout, or Flow).
  • Occlusion-based (PMask): Measuring how much a word's representation changes when another word is masked.
  • Gradient-based: Calculating word influence via gradients (e.g., Grad, GInput, GInteg).

These mined structures are represented as weighted graphs where importance scores become edge weights, allowing the model to focus on the most relevant relational connections.

Key Results and Insights

The models were evaluated on three diverse benchmarks: ACE05 (general news), SciERC (scientific abstracts), and WebNLG (crowdsourced triples).

F1 scores across ACE05, SciERC, and WebNLG datasets show that unsupervised structures (green) consistently outperform or match traditional dependency-based methods (orange/red).
  • Performance Gains: Mined structures improved F1 scores in 20 out of 21 experimental settings. Raw attention (Attn) was often the strongest performer, achieving the best results on ACE05 (+1.60 F1 over baseline) and SciERC (+0.84 F1).
  • Efficiency: Attention-based mining is significantly faster than traditional parsers like spaCy or UDPipe, offering up to an 8x speedup.
  • Domain Adaptability: On specialized scientific text (SciERC), traditional parsers often fail. However, by using a version of BERT pre-trained on scientific text (SciBERT), the unsupervised structures automatically adapted to the domain without manual re-annotation.
  • Noise Tolerance: While GCNs are generally robust, the specific structural information provided by PLMs offers a clear, non-random boost over both baseline and random noise structures.

Conclusion

Unsupervised structures mined from PLMs are not only faster to compute but often outperform traditional linguistic trees, especially in specialized domains. While choosing the optimal layer for extraction remains a subject for future study, these results suggest that internal PLM structures are a viable and powerful alternative to manual syntactic parsing for relation extraction.

Summary updated

Abstract

Syntactic trees have been widely applied in relation extraction (RE). However, since parsing qualities are not stable on different text domains and a pre-defined grammar may not well fit the target relation schema, the introduction of syntactic structures sometimes fails to improve RE performances consistently. In this work, we study RE models with various unsupervised structures mined from pre-trained language models (e.g., BERT). We show that, similar to syntactic trees, unsupervised structures are quite informative for RE task: they are able to obtain competitive (even the best) performance scores on benchmark RE datasets (ACE05, WebNLG, SciERC). We also conduct detailed analyses on their abilities of adapting new RE domains and influence of noise links in those structures. The results suggest that unsupervised structures are reasonable alternatives of commonly used syntactic structures in relation extraction models.
BibTeX
@inproceedings{yang2022explore,
  author = {Xi Yang and Tao Ji and Yuanbin Wu},
  title = {{Explore Unsupervised Structures in Pretrained Models for Relation Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=6XHSXXh917},
}
Figure