Abstract
Structural Knowledge for Information Extraction (SKIE) is a novel pre-training framework designed to bridge the gap between unstructured text and structured semantic representations. By leveraging automatically generated Abstract Meaning Representation (AMR) graphs and contrastive learning, SKIE aligns textual sequences with their underlying semantic "skeletons." This approach significantly improves performance in Named Entity Recognition (NER), Relation Extraction (RE), and Event Extraction (EE), particularly in low-resource and zero-shot settings.
Introduction
Information Extraction (IE) is the process of converting unstructured text into structured formats like tables or graphs. While Large Language Models (LLMs) have advanced the field, they often struggle with two primary challenges: the high cost of human-labeled data and "structural blindness." Most pre-training objectives focus on word-to-word patterns, neglecting the explicit structural logic—such as "who did what to whom"—that is often buried within raw text.
SKIE addresses these issues by utilizing "low-cost" supervision from automatically generated semantic graphs, providing a structural signal that guides the model toward a deeper understanding of relational grammar.
The SKIE Framework
The SKIE framework consists of three integrated modules designed to capture and align structural semantic knowledge.
Topology Enhancement
The model uses an off-the-shelf parser to transform text into Abstract Meaning Representation (AMR) graphs. To capture multi-level semantic importance, SKIE employs a k-core algorithm to extract cohesive subgraphs. This identifies the most densely connected semantic hubs, allowing the model to learn from diverse structural samples at various levels of complexity.
Encoding Cohesion
SKIE utilizes a dual-encoding strategy:
- Text Encoder: A RoBERTa-Large model processes the raw input strings.
- Graph Encoder (T-GSN): A Topology-aware Graph Substructure Network is proposed to preserve edge types and local substructures within AMR graphs, overcoming the limitations of standard Graph Convolutional Networks (GCNs).
Contrastive Learning
A triplet loss strategy is used to align the text and graph embeddings in a shared vector space. By pulling matching text-graph pairs closer and pushing non-matching pairs apart, the model learns the direct correspondence between a sentence's surface form and its structural logic.
Experimental Evaluation
The authors pre-trained SKIE on a unified unsupervised corpus of approximately 1.2 million instances spanning NER, RE, and EE. Evaluation was conducted across eight standard benchmarks, including ACE04, ACE05, and CoNLL03/04. Additionally, zero-shot robustness was tested using the CrossNER dataset across multiple domains such as AI, Literature, and Politics.
Results and Discussion
SKIE demonstrated superior performance across all evaluated tasks and benchmarks.
Key Findings
- General Performance: SKIE outperformed state-of-the-art baselines like UIE and USM, achieving average F1 improvements of 1.49% in NER, 6.75% in RE, and 3.42% in EE.
- Few-Shot and Zero-Shot Excellence: The model showed remarkable robustness in low-resource scenarios. For instance, in 1-shot Relation Extraction on CoNLL04, SKIE achieved an F1 of 37.54. In zero-shot settings on CrossNER, it reached an average F1 of 58.03, significantly surpassing competing models.
- Structural Awareness: Error analysis indicates that structural pre-training helps the model reason about the roles words play, reducing misclassifications (e.g., confusing persons with locations).
- Cross-Lingual Transfer: Interestingly, pre-training on English AMR graphs improved performance on other languages, suggesting that semantic structures possess a degree of cross-linguistic universality.
Conclusion
SKIE presents a powerful method for enhancing IE by integrating structural semantic knowledge without the need for intensive human annotation. While the framework faces challenges regarding computational cost and the noise inherent in automated parsing, it sets a new standard for structural awareness in pre-trained models. Future work will explore more efficient architectures and methods to further mitigate parsing noise.
Summary updated