Supervised Relation Classification as Two-way Span-Prediction

Amir David Nissan Cohen, Shachar Rosenman, Yoav GoldbergAKBC 2022 · OpenReview

TL;DR

This paper proposes a novel approach to relation classification (RC) by reframing it as a span-prediction problem, similar to question answering (QA). By converting RC datasets into a series of questions, their method achieves state-of-the-art results on several benchmarks, outperforming traditional embedding-based classification methods.

What Problem Does This Paper Address?

Relation Classification (RC) is a fundamental task in Natural Language Processing (NLP) that aims to identify semantic relationships between pairs of entities within a given text. Traditional RC models typically encode the sentence and entity pair into a single vector representation, which is then classified into one of several predefined relation types. However, the authors argue that this "classification view" is suboptimal. They point to prior work that highlights errors stemming from models not fully considering the marked entities or relying on shallow heuristics. The core problem is that existing methods might not effectively capture the nuanced semantic relationships required for accurate RC, especially when dealing with complex sentences or subtle relational cues.

Methods

The paper's central innovation is to transform the RC task into a span-prediction (SP) problem, drawing inspiration from Question Answering (QA) systems. Instead of directly classifying a relation, their approach converts each RC instance into one or more question-answer pairs. For a given relation and entity pair, they generate questions that, when answered by a span-prediction model, should yield the other entity or confirm its presence.

Key aspects of their method include:

  • Bidirectional Questions: For each relation, they generate two questions: one that predicts the tail entity given the head entity and relation, and another that predicts the head entity given the tail entity and relation. This "two-way" approach is shown to significantly improve performance.
  • Template Formulation: They explore various ways to formulate these questions, ranging from natural language questions (e.g., "Who was born in X?") to using the relation name directly, or even abstract "unique tokens."
  • Supervised Span Prediction: They train span-prediction models in a fully supervised manner, using the generated question-answer pairs derived from existing RC datasets. This contrasts with some prior QA-based RC work that focused on zero-shot or few-shot settings.
  • Answer Combination: When using bidirectional questions, they combine the answers using an "OR" operation, meaning if either question yields a correct span, the relation is considered present.

Datasets

The authors evaluate their approach on three widely used RC datasets:

  • TACRED: A large dataset with 41 relation types, commonly used for evaluating RC models.
  • SemEval 2010 Task 8: A smaller dataset with 9 abstract relation types, offering a different set of challenges.
  • Challenge Relation Extraction (CRE): A dataset specifically designed to expose shallow heuristics in RC models, making it a good testbed for robust methods.

Key Insights and Ideas

  • Span-Prediction is a More Natural Fit for RC: The paper strongly advocates that framing RC as span-prediction, rather than direct classification, leads to better performance. This is because SP models are forced to identify the exact spans that constitute the relation, mitigating issues with models ignoring entities or relying on superficial cues.
  • Bidirectional Questions Boost Performance: Using two questions per relation (one predicting head, one predicting tail) significantly improves results compared to using a single question. This suggests that capturing the relation from both directions provides a more comprehensive understanding.
  • Template Design Matters: The amount of semantic information encoded in the question templates correlates with model performance. More informative templates (like natural language questions) generally lead to better results, though there are some anomalies observed.
  • Supervised SP Outperforms Zero-Shot QA: Leveraging existing QA datasets (like SQuAD) in a zero-shot manner for RC yields high recall but harms overall accuracy. This highlights the importance of supervised training on RC-specific data for achieving top performance.

Key Results

The proposed span-prediction approach achieves state-of-the-art results on the evaluated datasets:

  • TACRED: Achieved a 2.3 F1 point improvement over the previous state-of-the-art.
  • SemEval: Achieved a 0.9 F1 point improvement over the previous state-of-the-art.
  • CRE: Their span-prediction models consistently outperformed traditional RC and QA models.
  • ALBERT-based models: Showed significant gains, outperforming the best TACRED model (KEPLER) by 2.3% F1, even when KEPLER used external data.

Limitations and Future Work

  • Computational Cost: The span-prediction method is noted as being more computationally expensive than traditional RC, requiring multiple contextualized embedding operations. The authors suggest improving computational efficiency as a direction for future work.
  • Template Anomaly: An unexpected finding was that on ALBERT models, the "relation name" reduction performed worse than the "token" reduction, which they currently have no convincing explanation for.
  • Generalization to Unseen Relations: While the paper focuses on supervised settings, the QA-inspired nature of the approach hints at potential for zero-shot or few-shot learning on unseen relations, which could be explored further.
  • Answer Combination Strategies: While "OR" combination was effective, exploring other strategies for combining answers from bidirectional questions could yield further improvements.

Summary updated

BibTeX
@inproceedings{cohen2022supervised,
  author = {Amir David Nissan Cohen and Shachar Rosenman and Yoav Goldberg},
  title = {{Supervised Relation Classification as Two-way Span-Prediction}},
  booktitle = {AKBC},
  year = {2022},
  url = {https://openreview.net/forum?id=OWMsma8yoH},
}
Figure