D-REX: Dialogue Relation Extraction with Explanations

Alon Albalak, Varun Embar, Yi-Lin Tuan, Lise Getoor, William Yang Wang

Abstract

Existing research studies on cross-sentence relation extraction in long-form multi-party conversations aim to improve relation extraction without considering the explainability of such methods. This work addresses that gap by focusing on extracting explanations that indicate that a relation exists while using only partially labeled data. We propose our model-agnostic framework, D-REX, a policy-guided semi-supervised algorithm that explains and ranks relations. We frame relation extraction as a re-ranking task and include relation- and entity-specific explanations as an intermediate step of the inference process. We find that about 90% of the time, human annotators prefer D-REX's explanations over a strong BERT-based joint relation extraction and explanation model. Finally, our evaluations on a dialogue relation extraction dataset show that our method is simple yet effective and achieves a state-of-the-art F1 score on relation extraction, improving upon existing methods by 13.5%.

Introduction

In traditional Natural Language Processing (NLP), Relation Extraction (RE) usually occurs within a single sentence. However, real-world dialogues are often spread across multiple turns and speakers, making it difficult for models to track entity relationships. Existing dialogue RE models often act as "black boxes," identifying relationships without providing interpretable logic. Furthermore, while some datasets include "trigger words" (specific phrases that prove a relation exists), most models do not use them effectively to improve performance or provide explanations to users.

The D-REX Framework

D-REX (Dialogue Relation Extraction with eXplanations) is a model-agnostic framework that treats relation extraction as a re-ranking task. The process is broken into three distinct steps:

  1. Initial Ranking ($R$): A standard model (like BERT or RoBERTa) provides a "best guess" ranking of possible relations.
  2. Explanation Extraction ($EX$): For the top-ranked relations, this module identifies a "trigger" span—the specific phrase in the dialogue that indicates the relation holds.
  3. Relation Re-Ranking ($RR$): This module takes the original dialogue plus the extracted explanation to produce a final, more accurate ranking.
The D-REX framework architecture, illustrating the flow from initial ranking to explanation extraction and final re-ranking.

Because labeling explanations is labor-intensive, D-REX uses a policy-guided semi-supervised approach. The $EX$ module is trained using Reinforcement Learning (RL) with two key rewards: a Re-ranking Reward (does the explanation help the re-ranker get the right answer?) and a Leave-one-out (LOO) Reward (does performance drop if the explanation is hidden?). This ensures the model identifies truly essential evidence.

Evaluation and Key Results

The study utilized DialogRE V2, a human-annotated dataset based on transcripts from the TV show Friends. It contains 1,788 dialogues and 36 relation types, with manually annotated trigger words for about 40% of the relational triples.

Performance Gains

D-REX significantly improved the accuracy of baseline models. As shown in the results table, D-REX improved F1 scores by 1.2% to 4.7% over standard BERT and RoBERTa implementations.

Performance comparison of D-REX against baseline models on the DialogRE dataset, showing improvements in F1 and MRR scores.

Human Interpretability

Beyond raw metrics, human evaluators were 4.2 times more likely to prefer D-REX’s explanations over those from a standard joint model. Surprisingly, annotators preferred D-REX's generated explanations over the "ground truth" human-annotated triggers 12% of the time, suggesting the model finds valid evidence that humans might overlook.

Discussion and Limitations

The paper demonstrates that separating relation prediction and explanation into a re-ranking pipeline boosts accuracy by ensuring the explanation actually supports the predicted relation. The "Leave-one-out" reward is particularly effective at teaching the model to identify the most "salient" or informative parts of a conversation.

However, limitations remain. The model was primarily tested on the Friends dataset in English; its performance in formal domains (medical, legal) or languages with complex inflections is yet to be determined. Additionally, while semi-supervised, the model's performance degrades when labeled triggers drop below 20% of the training data. Future work may involve using pre-trained span extraction models to further reduce manual label dependency and exploring generative natural language summaries.

Summary updated

Abstract

Existing research studies on cross-sentence relation extraction in long-form multi-party conversations aim to improve relation extraction without considering the explainability of such methods. This work addresses that gap by focusing on extracting explanations that indicate that a relation exists while using only partially labeled data. We propose our model-agnostic framework, D-REX, a policy-guided semi-supervised algorithm that explains and ranks relations. We frame relation extraction as a re-ranking task and include relation- and entity-specific explanations as an intermediate step of the inference process. We find that about 90% of the time, human annotators prefer D-REX's explanations over a strong BERT-based joint relation extraction and explanation model. Finally, our evaluations on a dialogue relation extraction dataset show that our method is simple yet effective and achieves a state-of-the-art F1 score on relation extraction, improving upon existing methods by 13.5%.
BibTeX
@inproceedings{albalak2022d,
  author = {Alon Albalak and Varun Embar and Yi-Lin Tuan and Lise Getoor and William Yang Wang},
  title = {{D-REX: Dialogue Relation Extraction with Explanations}},
  booktitle = {ConvAI @ ACL},
  year = {2022},
  url = {https://openreview.net/forum?id=euO5w6Pf5ym},
}

Loading related content...

Figure