TL;DR
The paper introduces $RE^2$, a framework for Relation Extraction (RE) that "thinks rationally" by filtering out noisy, irrelevant parts of a sentence. By using a novel rationale extraction module constrained by continuity and sparsity, the model identifies the most informative segments of text, leading to state-of-the-art performance and improved interpretability, especially in low-resource scenarios.
What Problem Does This Paper Address?
Relation Extraction (RE) is the task of identifying the relationship between two entities in a sentence (e.g., in "Ryan and Yaz knew each other from school," the relation between Ryan and Yaz is alumni).
The authors identify a major hurdle: contextual noise. Sentences often contain "distractor" words that mislead models. For example, if a sentence says "Ryan and Yaz... were similar to me and my friend," a standard model might latch onto the word "friend" and incorrectly predict the relation as peer instead of alumni.
Existing approaches like "Entity Thinking" (focusing on entity types) or "Counterfactual Thinking" (removing model bias) don't explicitly remove this noisy content. Furthermore, previous attempts at "Rationale Extraction" often produced fragmented, unreadable snippets of text that destroyed the semantic flow necessary for modern language models like BERT to function effectively.
Methods
The authors propose the Continuous Rationale Extractor, which applies an optimizable binary mask to every token in a sentence. To ensure these masks are actually useful, they introduce two primary constraints via a factor graph:
- Continuity (CON): This pairwise factor encourages the model to select consecutive tokens. By favoring sequences over isolated words, the extracted "rationale" remains fluent and semantically coherent, which is vital for the downstream BERT-based classifier.
- Sparsity (SPA): This factor imposes a limit ($K$) on the total number of tokens selected. This forces the model to be "rational" and discard irrelevant segments, keeping only the most critical information.
The Training Challenge: A significant hurdle is that there are no "gold labels" for what a good rationale looks like. To solve this, the authors treat rationale selection as a latent variable problem. They use Marginal Inference with Lagrange Multipliers and a sampling strategy called Perturb-and-MAP to make the selection process differentiable. This allows the rationale extractor and the relation classifier to be trained together end-to-end.
Datasets
The model was evaluated on four standard RE benchmarks:
- SemEval-2010 Task 8: A classic dataset with 19 relation types.
- TACRED: A large-scale dataset from Stanford with 42 relation types.
- TACRED-Revisit: A cleaned version of TACRED addressing label noise.
- Re-TACRED: A more recent, highly refined version of the TACRED dataset.
The authors also tested the model in low-resource settings, randomly sampling 10%, 25%, and 50% of the training data to test robustness.
Key Insights and Ideas
- Rationales as Regularizers: In low-resource settings (10% data), $RE^2$ showed a 1.1% F1 improvement, compared to 0.9% in full-data settings. This suggests that forcing the model to find rationales acts as a powerful regularizer, preventing it from over-fitting to spurious correlations in small datasets.
- The "Aha" Moment on Continuity: The authors found that "broken" rationales (isolated words) significantly hurt performance. By enforcing continuity, the perplexity of the extracted text (measured by GPT-3) was nearly as low as the original sentences, proving that the model preserves natural language structure.
- Optimal Sparsity: Through ablation, they discovered that the "sweet spot" for information density is around 60%. Selecting too few tokens loses context, while selecting too many reintroduces the noise they were trying to avoid.
Key Results
- State-of-the-Art Performance: $RE^2$ outperformed previous SOTA models (like UNIREX and CORE) across all four datasets.
- Human Validation: In a human study, annotators were asked to predict relations using only the extracted rationales. Humans achieved higher accuracy using the rationales (95.5% on SemEval) than they did using the original, full sentences (94.3%), confirming that the model successfully removed confusing noise.
- Interpretability: Unlike "black box" models, $RE^2$ provides a clear visual mask of what it "saw" to make its decision, making the model's logic transparent to users.
Limitations and Future Work
The authors acknowledge that while the model is robust, the sparsity parameter ($K$) is a hyperparameter that currently requires manual tuning for different datasets.
Future work includes:
- Extending the framework to the construction of Knowledge Graphs.
- Applying continuous rationale extraction to other Information Retrieval tasks like document ranking or question answering.
- Exploring more automated ways to determine the optimal sparsity for a given sentence length or complexity.
Summary updated