Conversational Response Re-ranking Based on Event Causality and Role Factored Tensor Event Embedding

Shohei Tanaka, Koichiro Yoshino, Katsuhito Sudoh, Satoshi Nakamura

TL;DR

This paper introduces a novel re-ranking framework for neural conversational models that selects responses based on logical event causality (e.g., recognizing that "being exhausted" should naturally lead to "relaxing"). By leveraging a Role Factored Tensor Model (RFTM) to embed predicate-argument structures into a continuous space, the system robustly matches dialogue history to a large database of causal relations, significantly improving conversational diversity and continuity.

What Problem Does This Paper Address?

Standard Neural Conversational Models (NCMs), such as standard Encoder-Decoder architectures, frequently suffer from the "safe response" problem. Because they are trained to maximize token likelihood, they tend to output generic, dull phrases like "I don't know" or "OK."

While modern architectures try to incorporate broader context, they lack an explicit understanding of how real-world events logically connect over time. To generate truly engaging and coherent conversations, a system needs to understand event causality—the idea that an effect event (like "relieving stress") is highly likely to follow a cause event (like "being stressed out"). The challenge lies in effectively extracting these event relations from text and matching them to ongoing conversations, especially when the exact words used in the dialogue haven't been seen in the training corpus.

Methods

The authors tackle this problem using a four-stage re-ranking pipeline:

  1. Candidate Generation: An NCM generates the $N$-best response candidates using beam search or sampling.
  2. Event Parsing: The system extracts events—defined as predicate-argument structures (e.g., predicate: relieve, accusative: stress)—from both the dialogue history and the response candidates using the Kurohashi Nagao Parser (KNP).
  3. Distributed Event Embedding: To avoid the limitations of exact string matching, the authors utilize the Role Factored Tensor Model (RFTM). Arguments are mapped via Skip-gram, and predicates are embedded based on their case frames (nominative, accusative, dative, etc.). A 3D tensor then composes these into a single vector representing the event context.
  4. Causality Re-ranking: The system computes a modified lift score (a measure of causality strength) between the history events and candidate response events using cosine similarity in the embedding space. The final response is chosen by balancing the NCM's generation probability with this causality score.
Figure 1: Neural conversational model+re-ranking using event causality; a response that has an event causality relation (“be exhausted” →“relax”) to the dialogue history is selected by the re-ranking.

Datasets

The models were trained and evaluated using several large-scale Japanese corpora:

  • Dialogue Corpus: 2,632,114 dialogues extracted from Japanese Twitter, split into training, validation, and testing sets.
  • Event Causality Pairs: A pre-extracted pool of 420,000 cause-effect event pairs derived from a massive 1.6 billion text corpus based on co-occurrence statistics.
  • Embedding Data: A Japanese Wikipedia dump was used to train the initial Skip-gram and predicate embeddings, while the Mainichi Newspaper dataset 2017 was used to fine-tune the RFTM.

Key Insights and Ideas

  • The Sparsity Trap: Relying on exact string matching for event causality pairs is incredibly restrictive. In their baseline "Re-ranking" model, only about 7% to 12% of sentences actually triggered a causal match from the database.
  • Embedding to the Rescue: By introducing distributed representations (RFTM), the system can generalize causal relationships. For example, even if "be exhausted $\rightarrow$ relax" isn't explicitly in the database, the system can map it to the known pair "be stressed out $\rightarrow$ relieve stress" via vector similarity. This skyrocketed the re-ranking trigger rate to over 71%.
  • The Metric Trade-off: Introducing causality re-ranking actually lowers standard overlap metrics like BLEU and NIST. This is an insightful reminder that BLEU penalizes diverse, creative responses that deviate from a single ground-truth reference, even if the new response is logically superior.

Key Results

The authors evaluated their approach using both automatic metrics and human crowdsourcing:

  • Automatic Evaluation: The embedding-based re-ranking model (Re-ranking (emb)) achieved the highest distinct-2 (vocabulary diversity) and Pointwise Mutual Information (PMI) scores, confirming that the selected responses were both more diverse and more contextually relevant.
Quantitative comparison of automatic evaluation metrics before and after re-ranking
  • Human Evaluation: In side-by-side comparisons, crowd-workers found that the proposed embedding method significantly improved dialogue continuity (making the system easier to respond to) compared to the baseline 1-best selection.
Table 4: 1-best v.s. Re-ranking; # dialogues is 100.

Limitations and Future Work

Despite the improvements, the authors highlight two primary challenges:

  1. Over-Generalization: Because the embedding space groups similar concepts, it occasionally creates false causal links. For instance, it generalized the valid causality "enter restaurant $\rightarrow$ order beer" into the nonsensical dialogue relation "drink alcohol $\rightarrow$ can drink alcohol."
  2. Naturalness vs. Coherency: The re-ranking function heavily prioritizes word-level topical coherency, which sometimes inadvertently selects responses that feel grammatically stiff or unnatural.

Future work will focus on optimizing the similarity thresholds within the embedding space to strike a better balance between logical continuity and conversational naturalness.

Summary updated

Abstract

We propose a novel method for selecting coherent and diverse responses for a given dialogue context. The proposed method re-ranks response candidates generated from conversational models by using event causality relations between events in a dialogue history and response candidates (e.g., ``be stressed out'' precedes ``relieve stress''). We use distributed event representation based on the Role Factored Tensor Model for a robust matching of event causality relations due to limited event causality knowledge of the system. Experimental results showed that the proposed method improved coherency and dialogue continuity of system responses.
BibTeX
@inproceedings{tanaka2019conversational,
  author = {Shohei Tanaka and Koichiro Yoshino and Katsuhito Sudoh and Satoshi Nakamura},
  title = {{Conversational Response Re-ranking Based on Event Causality and Role Factored Tensor Event Embedding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure