Generic and Trend-aware Curriculum Learning for Relation Extraction

Nidhi Vakil, Hadi Amiri

Abstract

We present a generic and trend-aware curriculum learning approach that effectively integrates textual and structural information in text graphs for relation extraction between entities, which we consider as node pairs in graphs. The proposed model extends existing curriculum learning approaches by incorporating sample-level loss trends to better discriminate easier from harder samples and schedule them for training. The model results in a robust estimation of sample difficulty and shows sizable improvement over the state-of-the-art approaches across several datasets.

Introduction

Relation extraction (RE) is a fundamental task in natural language processing, aimed at identifying meaningful connections between entities, such as gene-disease associations. While Graph Neural Networks (GNNs) have become a standard tool for modeling these relationships by treating entities as nodes, they face two significant challenges. First, the iterative message-passing process in GNNs often leads to the "washing out" of critical textual information. Second, standard training regimes treat all samples equally, regardless of their inherent difficulty, which can cause models to struggle with complex topologies or subtle textual cues.

Methodology

To address these challenges, we propose a novel framework consisting of two main components: the Graph Text Neural Network (GTNN) and the Trend-aware SuperLoss (Trend-SL).

Graph Text Neural Network (GTNN)

Traditional GNNs typically use text only for node feature initialization. In contrast, GTNN preserves textual nuances by "short-circuiting" the data. It feeds raw text embeddings and Information Retrieval (IR) scores (e.g., BM25) directly into the final prediction layer, ensuring that the graph structure supplements rather than replaces the rich textual information.

Trend-aware SuperLoss (Trend-SL)

Curriculum learning typically ranks samples by their instantaneous loss. Trend-SL improves upon this by analyzing the loss trajectory over multiple training iterations. A falling trend indicates a sample is becoming "easy," warranting higher weight, while a rising trend flags a "hard" or confusing sample that should be temporarily de-prioritized.

The GTNN architecture integrates graph neural encoders with textual summary encoders, supervised by the Trend SuperLoss during training.

Experimental Results

The model was evaluated on three datasets: GDPR (a challenging new gene-disease-phenotype dataset), PGR (a PubMed-derived benchmark), and a Wikipedia-based link prediction dataset.

Performance Gains

The GTNN architecture alone outperformed baseline GNNs by an average of 8.6 F1 points. The addition of the Trend-SL curriculum provided a further 0.7 point boost, achieving an average F1 of 89.9 across all datasets. Notably, on the PGR dataset, the model reached an F1 of 94.2, surpassing BioBERT-based models.

Training Stability

A key advantage of the trend-aware approach is the reduction of "inversions"—instances where a sample's classification as "easy" or "hard" fluctuates rapidly between epochs. Trend-SL demonstrates significantly higher stability compared to standard SuperLoss (SL).

Trend-SL significantly reduces the fraction of inversions compared to standard SuperLoss, leading to more stable training.

Discussion and Conclusion

Our findings emphasize that for relation extraction, graph structures are most effective when they complement direct textual features. The "aha moment" of this research is that a sample's learning curve is a more robust indicator of difficulty than its current error. While calculating loss trends adds some computational complexity, the resulting stability and performance gains are substantial. Future work will explore applying these trend-aware techniques to other graph-based tasks like node classification and graph-to-text generation.

Summary updated

Abstract

We present a generic and trend-aware curriculum learning approach that effectively integrates textual and structural information in text graphs for relation extraction between entities, which we consider as node pairs in graphs. The proposed model extends existing curriculum learning approaches by incorporating sample-level loss trends to better discriminate easier from harder samples and schedule them for training. The model results in a robust estimation of sample difficulty and shows sizable improvement over the state-of-the-art approaches across several datasets.
BibTeX
@inproceedings{vakil2022generic,
  author = {Nidhi Vakil and Hadi Amiri},
  title = {{Generic and Trend-aware Curriculum Learning for Relation Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=VThi57EKId},
}
Figure