Abstract
Relation extraction (RE) is a fundamental task in information retrieval that aims to identify relationships between entities within a sentence. However, standard models often struggle with contextual noise—irrelevant words that mislead the reasoning process. We introduce $RE^2$, a novel framework that leverages continuous rationale extraction to filter out noisy segments while preserving semantic coherence. By applying an optimizable binary mask constrained by continuity and sparsity, $RE^2$ identifies the most informative text segments for relation classification. Our experiments across four major benchmarks demonstrate that $RE^2$ achieves state-of-the-art performance, particularly in low-resource scenarios, while providing human-interpretable explanations for its predictions.
Introduction: The Challenge of Contextual Noise
The primary goal of Relation Extraction (RE) is to determine the relationship between two entities based on their context. For instance, in the sentence "Ryan and Yaz knew each other from school," the relation between Ryan and Yaz is alumni.
A significant hurdle in this task is contextual noise. Sentences often contain "distractor" words that lead models astray. Consider a sentence like: "Ryan and Yaz... were similar to me and my friend." A standard model might focus on the word "friend" and incorrectly predict the relation as peer instead of alumni.
Existing methods, such as "Entity Thinking" (focusing on entity types) or "Counterfactual Thinking" (removing model bias), do not explicitly discard this noisy content. Furthermore, previous attempts at rationale extraction often resulted in fragmented, unreadable snippets that destroyed the semantic flow necessary for modern language models like BERT to function effectively.
The $RE^2$ Framework
To address these issues, we propose the Continuous Rationale Extractor. This module applies an optimizable binary mask to every token in a sentence. To ensure the extracted rationales are useful and readable, we introduce two primary constraints via a factor graph:
- Continuity (CON): This factor encourages the model to select consecutive tokens. By favoring sequences over isolated words, the extracted rationale remains fluent and semantically coherent, which is vital for maintaining the performance of the downstream BERT-based classifier.
- Sparsity (SPA): This factor imposes a limit ($K$) on the total number of tokens selected. This forces the model to be "rational" and discard irrelevant segments, keeping only the most critical information.
Training via Latent Variables
Because "gold labels" for rationales do not exist, we treat rationale selection as a latent variable problem. We utilize Marginal Inference with Lagrange Multipliers and a sampling strategy called Perturb-and-MAP to make the selection process differentiable. This allows the rationale extractor and the relation classifier to be trained together end-to-end.
Experimental Evaluation
We evaluated $RE^2$ on four standard RE benchmarks: SemEval-2010 Task 8, TACRED, TACRED-Revisit, and Re-TACRED. To test robustness, we also conducted experiments in low-resource settings, using only 10%, 25%, and 50% of the training data.
Key Results
- State-of-the-Art Performance: $RE^2$ outperformed previous SOTA models, including UNIREX and CORE, across all four datasets.
- Regularization in Low-Resource Settings: In the 10% data setting, $RE^2$ showed a 1.1% F1 improvement compared to 0.9% in full-data settings. This suggests that forcing the model to find rationales acts as a powerful regularizer, preventing it from over-fitting to spurious correlations.
- Human Validation: In a human study, annotators achieved higher accuracy using only the extracted rationales (95.5% on SemEval) than they did using the original, full sentences (94.3%). This confirms that the model successfully isolates the most relevant information while removing confusing noise.
- Interpretability: Unlike "black box" models, $RE^2$ provides a clear visual mask of the evidence it used to make a decision, making the logic transparent to users.
Conclusion and Future Work
$RE^2$ demonstrates that "thinking rationally" by filtering noise through continuous and sparse rationale extraction significantly improves both the performance and interpretability of relation extraction models. While the sparsity parameter ($K$) currently requires manual tuning, future work will explore automated methods for determining optimal sparsity based on sentence complexity. We also plan to extend this framework to Knowledge Graph construction and other information retrieval tasks such as document ranking and question answering.
Summary updated