Integrating Structural Semantic Knowledge for Enhanced Information Extraction Pre-training

Xiaoyang Yi, Yuru Bao, Jian Zhang, Yifang Qin, Faxin Lin

Abstract

Structural Knowledge for Information Extraction (SKIE) is a novel pre-training framework designed to bridge the gap between unstructured text and structured semantic representations. By leveraging automatically generated Abstract Meaning Representation (AMR) graphs and contrastive learning, SKIE aligns textual sequences with their underlying semantic "skeletons." This approach significantly improves performance in Named Entity Recognition (NER), Relation Extraction (RE), and Event Extraction (EE), particularly in low-resource and zero-shot settings.

Introduction

Information Extraction (IE) is the process of converting unstructured text into structured formats like tables or graphs. While Large Language Models (LLMs) have advanced the field, they often struggle with two primary challenges: the high cost of human-labeled data and "structural blindness." Most pre-training objectives focus on word-to-word patterns, neglecting the explicit structural logic—such as "who did what to whom"—that is often buried within raw text.

SKIE addresses these issues by utilizing "low-cost" supervision from automatically generated semantic graphs, providing a structural signal that guides the model toward a deeper understanding of relational grammar.

The SKIE Framework

The SKIE framework consists of three integrated modules designed to capture and align structural semantic knowledge.

Overview of the SKIE framework, illustrating the integration of text and graph encoding via contrastive learning.

Topology Enhancement

The model uses an off-the-shelf parser to transform text into Abstract Meaning Representation (AMR) graphs. To capture multi-level semantic importance, SKIE employs a k-core algorithm to extract cohesive subgraphs. This identifies the most densely connected semantic hubs, allowing the model to learn from diverse structural samples at various levels of complexity.

Encoding Cohesion

SKIE utilizes a dual-encoding strategy:

  • Text Encoder: A RoBERTa-Large model processes the raw input strings.
  • Graph Encoder (T-GSN): A Topology-aware Graph Substructure Network is proposed to preserve edge types and local substructures within AMR graphs, overcoming the limitations of standard Graph Convolutional Networks (GCNs).

Contrastive Learning

A triplet loss strategy is used to align the text and graph embeddings in a shared vector space. By pulling matching text-graph pairs closer and pushing non-matching pairs apart, the model learns the direct correspondence between a sentence's surface form and its structural logic.

Experimental Evaluation

The authors pre-trained SKIE on a unified unsupervised corpus of approximately 1.2 million instances spanning NER, RE, and EE. Evaluation was conducted across eight standard benchmarks, including ACE04, ACE05, and CoNLL03/04. Additionally, zero-shot robustness was tested using the CrossNER dataset across multiple domains such as AI, Literature, and Politics.

Results and Discussion

SKIE demonstrated superior performance across all evaluated tasks and benchmarks.

Performance comparison of SKIE against state-of-the-art models across NER, RE, and EE tasks.

Key Findings

  • General Performance: SKIE outperformed state-of-the-art baselines like UIE and USM, achieving average F1 improvements of 1.49% in NER, 6.75% in RE, and 3.42% in EE.
  • Few-Shot and Zero-Shot Excellence: The model showed remarkable robustness in low-resource scenarios. For instance, in 1-shot Relation Extraction on CoNLL04, SKIE achieved an F1 of 37.54. In zero-shot settings on CrossNER, it reached an average F1 of 58.03, significantly surpassing competing models.
  • Structural Awareness: Error analysis indicates that structural pre-training helps the model reason about the roles words play, reducing misclassifications (e.g., confusing persons with locations).
  • Cross-Lingual Transfer: Interestingly, pre-training on English AMR graphs improved performance on other languages, suggesting that semantic structures possess a degree of cross-linguistic universality.

Conclusion

SKIE presents a powerful method for enhancing IE by integrating structural semantic knowledge without the need for intensive human annotation. While the framework faces challenges regarding computational cost and the noise inherent in automated parsing, it sets a new standard for structural awareness in pre-trained models. Future work will explore more efficient architectures and methods to further mitigate parsing noise.

Summary updated

Abstract

Information Extraction (IE), aiming to extract structured information from unstructured natural language texts, can significantly benefit from pre-trained language models. However, existing pre-training methods solely focus on exploiting the textual knowledge, relying extensively on annotated large-scale datasets, which is labor-intensive and thus limits the scalability and versatility of the resulting models. To address these issues, we propose SKIE, a novel pre-training framework tailored for IE that integrates structural semantic knowledge via contrastive learning, effectively alleviating the annotation burden. Specifically, SKIE utilizes Abstract Meaning Representation (AMR) as a low-cost supervision source to boost model performance without human intervention. By enhancing the topology of AMR graphs, SKIE derives high-quality cohesive subgraphs as additional training samples, providing diverse multi-level structural semantic knowledge. Furthermore, SKIE refines the graph encoder to better capture cohesive information and edge relation information, thereby improving the pre-training efficacy. Extensive experimental results demonstrate that SKIE outperforms state-of-the-art baselines across multiple IE tasks and showcases exceptional performance in few-shot and zero-shot settings.
BibTeX
@inproceedings{yi2024integrating,
  author = {Xiaoyang Yi and Yuru Bao and Jian Zhang and Yifang Qin and Faxin Lin},
  title = {{Integrating Structural Semantic Knowledge for Enhanced Information Extraction Pre-training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.129},
}
Figure