Modeling Multi-Granularity Hierarchical Features for Relation Extraction

Xinnian Liang, Shuangzhi Wu, Mu Li, Zhoujun Li

Introduction

Relation Extraction (RE) is a fundamental task in Natural Language Processing, focused on identifying the semantic relationship between two target entities within a given text. Traditionally, state-of-the-art RE models have relied heavily on external knowledge sources—such as knowledge graphs or linguistic dependency trees—to provide structural context. However, these external dependencies introduce significant computational overhead and often inject "noise" due to errors in pre-training or parsing.

In this work, researchers propose the SMS framework, which shifts the focus back to the input text itself. By modeling multi-granularity hierarchical features at the sentence, mention, and segment levels, the SMS model demonstrates that deep internal feature extraction can outperform complex models reliant on external knowledge.

The SMS Framework

The core of the SMS approach is a Multi-Granularity Hierarchical Feature Extractor. Rather than treating the sentence as a flat sequence of tokens, the model processes information through three distinct hierarchical stages:

  1. Mention-Level Attention: Beyond the primary target entities, the model employs an attention mechanism to identify other mentions or co-references of those entities within the sentence. This ensures that all relevant entity-related signals are captured.
  2. Mention-Aware Segment Attention: This stage addresses the "middle-ground" of linguistic structure. Using Convolutional Neural Networks (CNNs), the model extracts n-gram segments (sequences of 1, 2, or 3 words). It then uses the previously identified mention information to "attend" to the most relevant segments. For example, in the sentence "Steve Jobs, the visionary co-founder of Apple, passed away," the model uses the entities to focus on the segment "co-founder of."
  3. Global Semantic Attention: Finally, the model generates a high-level representation of the entire sentence. It uses the combined entity and mention data as a query to extract the overarching context necessary for final classification.

This hierarchical architecture is designed to be modular, allowing it to be integrated into various base encoders such as LSTM or BERT-based architectures.

Experimental Results

The SMS model was evaluated against three major benchmarks: SemEval 2010 Task 8, TACRED, and TACRED Revisited. The results consistently showed that the hierarchical approach provides a significant performance boost over baseline models and knowledge-enhanced competitors.

  • TACRED: When integrated with a standard LSTM, SMS improved the F1 score by 5.5 points. When paired with SpanBERT-large, it achieved a state-of-the-art F1 score of 71.9.
  • TACRED Revisited: On this cleaned version of the dataset, the model reached an F1 of 79.8, notably outperforming models that utilize complex dependency tree information.
  • SemEval 2010: The SpanBERT-large+SMS configuration achieved an F1 of 90.3, demonstrating the model's efficacy even on smaller datasets.

Key Insights

The success of the SMS framework offers several critical insights into the nature of Relation Extraction:

  • The Importance of Segments: Ablation studies revealed that segment-level features (n-grams) are the primary drivers of performance gains. These segments act as the "missing link" between specific entity tokens and the broad sentence context.
  • Hierarchical Processing: The researchers found that treating features as a hierarchy—where mention data guides the search for relevant segments—mimics the human cognitive process of scanning a sentence for context clues relative to specific subjects.
  • Internal vs. External Knowledge: The fact that SMS outperformed models like LUKE or KnowBERT suggests that well-structured internal features can be more effective than external knowledge that may not be perfectly aligned with the target text.

Conclusion and Future Directions

The SMS framework proves that sophisticated, multi-granularity modeling of the input text can eliminate the need for noisy external knowledge in Relation Extraction. While highly effective for single-sentence, single-pair extraction, the authors note that future work will focus on extending this hierarchical approach to Document-Level RE and Multi-Relation Extraction, where multiple entity pairs and long-range dependencies present additional challenges. Furthermore, the team aims to investigate whether attention mechanisms can be further refined to simulate the specific structural benefits of dependency trees without the associated computational costs.

Summary updated

Abstract

Relation extraction is a key task in Natural Language Processing (NLP), which aims to extract relations between entity pairs from given texts. Recently, relation extraction (RE) has achieved remarkable progress with the development of deep neural networks. Most existing research focuses on constructing explicit structured features using external knowledge such as knowledge graph and dependency tree. In this paper, we propose a novel method to extract multi-granularity features based solely on the original input sentences. We show that effective structured features can be attained even without external knowledge. Three kinds of features based on the input sentences are fully exploited, which are in entity mention level, segment level, and sentence level. All the three are jointly and hierarchically modeled. We evaluate our method on three public benchmarks: SemEval 2010 Task 8, Tacred, and Tacred Revisited. To verify the effectiveness, we apply our method to different encoders such as LSTM and BERT. Experimental results show that our method significantly outperforms existing state-of-the-art models that even use external knowledge. Extensive analyses demonstrate that the performance of our model is contributed by the capture of multi-granularity features and the model of their hierarchical structure. Code and data are available at \url{https://github.com/xnliang98/sms}.
BibTeX
@inproceedings{liang2022modeling,
  author = {Xinnian Liang and Shuangzhi Wu and Mu Li and Zhoujun Li},
  title = {{Modeling Multi-Granularity Hierarchical Features for Relation Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.375},
}
Figure