Abstract: Relation Extraction (RE) identifies relations between entities in text, typically relying on supervised models that demand abundant high-quality data. Various approaches, including Data Augmentation (DA), have been proposed as promising solutions for addressing low-resource challenges in RE. However, existing DA methods in RE often struggle to ensure consistency and contextual diversity in generated data due to the fine-grained nature of RE. Inspired by the extensive generative capabilities of large language models (LLMs), we introduce a novel framework named ConsistRE, aiming to maintain context consistency in RE. ConsistRE initiates by collecting a substantial corpus from external resources and employing statistical algorithms and semantics to identify keyword hints closely related to relation instances. These keyword hints are subsequently integrated as contextual constraints in sentence generation, ensuring the preservation of relation dependence and diversity with LLMs. Additionally, we implement syntactic dependency selection to enhance the syntactic structure of the generated sentences. Experimental results from the evaluation of SemEval, TACRED, and TACREV datasets unequivocally demonstrate that ConsistRE outperforms other baselines in F1 values by 1.76%, 3.92%, and 2.53%, respectively, particularly when operating under low-resource experimental conditions.
The Challenge of Fine-Grained Relation Extraction
Relation Extraction (RE) is the task of identifying semantic relationships between entities in a sentence. For example, in the sentence "Thomas Edison was the inventor of the lightbulb," a model should identify a relationship between Thomas Edison (Subject) and lightbulb (Object) as inventor.
Training effective RE models requires significant amounts of labeled data. While Data Augmentation (DA) is a common solution, RE presents a unique challenge due to its fine-grained dependency. Existing DA methods often fail in one of two ways:
- Editing Methods: These perform simple operations like synonym swapping. While they preserve the original relation, they lack the contextual diversity needed for models to generalize.
- Generative Methods: These use models to generate new sentences. While diverse, they often suffer from "hallucinations" where the core relationship is accidentally altered (e.g., changing a "date of death" to a "date of birth"), introducing noise into the training set.
The ConsistRE Framework
To address these issues, we propose ConsistRE, a framework designed to transform Large Language Models (LLMs) into high-quality data generators that maintain strict context consistency. The pipeline consists of three main stages:
1. Keyword Hints Retrieval
Rather than allowing the LLM to generate context freely, ConsistRE first searches external resources (the internet) for sentences containing the target entity pair and relation. It then applies a scoring formula—integrating Pointwise Mutual Information (PMI), TF-IDF, and Semantic Similarity—to identify specific "keyword hints" that strongly signal the target relation.
2. Constrained Sentence Generation
These keyword hints serve as anchors for the LLM (e.g., GPT-3.5-turbo). By providing these hints as hard constraints in the prompt, the LLM is forced to generate sentences that are semantically tied to the correct relation. This process is further enhanced by In-Context Learning (ICL), where the model is shown high-quality examples from an example database before generation.
3. Syntactic Selection
To ensure the generated data is grammatically sound and structurally coherent, ConsistRE employs a filtering step based on Tree Edit Distance (TDS). It compares the syntactic dependency trees of the generated sentences against original examples, selecting only those that exhibit superior syntactic structures.
Experimental Evaluation
We evaluated ConsistRE on three major RE benchmarks: SemEval 2010 Task 8, TACRED, and TACREV. To simulate real-world constraints, we focused on low-resource settings, testing performance with as few as 5 to 100 "shots" (labeled examples) per relation.
Key Results
ConsistRE consistently outperformed traditional editing methods and state-of-the-art generative baselines. Using the ReDMP model as a backbone, ConsistRE achieved F1 score improvements of 1.48% on SemEval, 5.48% on TACRED, and 3.16% on TACREV over the strongest baselines. These gains were most significant in the most data-scarce environments (5-shot and 10-shot settings).
Insights on Constraints
Our analysis revealed a "Goldilocks zone" for keyword constraints. While keyword hints are essential—improving performance by over 5% compared to unconstrained generation—adding too many hints can be counterproductive. A single, high-quality keyword hint provides the best balance between maintaining relation consistency and allowing the LLM enough creative freedom to produce diverse training examples.
Conclusion
ConsistRE demonstrates that LLMs can be highly effective data augmentors for fine-grained tasks like Relation Extraction when properly constrained. By combining external knowledge retrieval with syntactic filtering, the framework produces synthetic data that is both diverse and relationally accurate, offering a robust solution for low-resource NLP challenges.
Summary updated