Conversational Response Re-ranking Based on Event Causality and Role Factored Tensor Event Embedding

Shohei Tanaka, Koichiro Yoshino, Katsuhito Sudoh, Satoshi Nakamura

TL;DR

This paper proposes a novel method to improve conversational AI by re-ranking candidate responses based on event causality. By leveraging event causality relations and a sophisticated event embedding technique, the authors demonstrate improved coherence and dialogue continuity in generated responses, addressing a key limitation of current neural conversational models.

What Problem Does This Paper Address?

Modern neural conversational models (NCMs) often struggle to generate coherent and engaging responses, even when presented with a dialogue history. This is largely due to their limited ability to deeply understand and utilize the causal relationships between events described in the conversation. While NCMs can produce grammatically correct sentences, they may lack the logical flow and contextual relevance that makes human conversation natural. This paper tackles the problem of selecting the best response from a set of candidates generated by an NCM, aiming to enhance the coherence and continuity of the dialogue by explicitly incorporating event causality.

Methods

The proposed method involves a four-part process:

  1. N-best Response Candidate Generation: Standard NCMs (specifically, an Encoder-Decoder with Attention and a Hierarchical Recurrent Encoder-Decoder) are used to generate multiple candidate responses for a given dialogue history.
  2. Event Extraction: An event parser (Kawahara Nagao Parser - KNP) is employed to extract predicate-argument structures (events) from both the dialogue history and the candidate responses.
  3. Distributed Event Representation: To overcome the limitations of relying solely on pre-defined event causality pairs, the authors introduce a distributed event representation using the Role Factored Tensor Model (RFTM). This model embeds events (predicates and their arguments) into fixed-length vectors, allowing for robust matching of similar events even if they aren't explicitly listed in the causality pool. The RFTM learns to represent the meaning of an event within its context.
  4. Re-ranking: A re-ranking score is calculated for each candidate response. This score combines the response's probability from the NCM with a measure of event causality between the dialogue history and the candidate response. The causality score is enhanced by using the RFTM embeddings to calculate similarities between events. This allows the system to identify responses that have a likely cause-effect relationship with the preceding dialogue.

Datasets

  • Training Data:
    • Japanese data from a Wikipedia dump was used for training Skip-gram and predicate word embeddings for the RFTM.
  • The Maichichi newspaper dataset 2017 was used for training the RFTM.
  • A large corpus of 2,632,114 dialogues from Japanese microblogs (Twitter) was used for training and testing the dialogue models. This corpus was split into training, validation, and testing sets.
  • Event Causality Pairs: 420,000 event causality pairs were extracted from a large-scale corpus (1.6 billion texts), based on co-occurring statistics and case frames.

Key Insights and Ideas

  • Event Causality as a Key to Coherence: The core idea is that understanding and leveraging the causal links between events in a dialogue is crucial for generating natural and coherent responses.
  • Bridging the Gap with Embeddings: Relying solely on explicit event causality pairs is insufficient. The RFTM-based distributed event representation allows for generalization and matching of events that are semantically similar but not identical, significantly improving the robustness of the causality matching.
  • Re-ranking for Improved Selection: Re-ranking candidate responses based on causality provides a mechanism to filter out less relevant or dull responses, even if they were initially assigned a high probability by the NCM.
  • Trade-off between Coherency and Naturalness: The study highlights a potential trade-off. While event embedding can improve causality matching, it might lead to over-generalization and less natural responses. Conversely, focusing solely on explicit causality might miss nuanced connections.

Key Results

  • Automatic Metrics:
    • Re-ranking (both with and without embedding) improved diversity metrics (dist-1, dist-2, PMI) compared to the 1-best NCM output.
  • The "Re-ranking (emb)" method (using event embedding) showed the highest dist-1, dist-2, and PMI scores.
  • Re-ranking generally lowered similarity-to-reference metrics (BLEU, NIST, extrema), as expected, since the original top-ranked responses were optimized for similarity.
  • Human Evaluation:
    • Word coherency improved with re-ranking without embedding but slightly lowered with embedding. This suggests human evaluators recognized explicit causality but struggled with generalized causality.
  • Dialogue continuity significantly improved with the "Re-ranking (emb)" method, likely due to the reduction of dull responses.

Limitations and Future Work

  • Over-generalization: The event embedding method can sometimes over-generalize events, leading to less natural or relevant causality relations (e.g., "drink alcohol" preceding "can drink alcohol" in Conversation 4).
  • Focus on Word Coherency vs. Naturalness: The current method prioritizes word coherency but may sacrifice response naturalness, as seen in some examples where responses became unnatural.
  • Future Work:
    • Preventing over-generalization by improving the embedding architecture.
  • Developing methods to maintain response naturalness while enhancing word choice coherency.
  • Investigating better thresholds for event embedding to balance coherency and continuity.
  • The proposed method is applicable to any language with a semantic parser due to its predicate-argument structure focus.

Summary updated

Abstract

We propose a novel method for selecting coherent and diverse responses for a given dialogue context. The proposed method re-ranks response candidates generated from conversational models by using event causality relations between events in a dialogue history and response candidates (e.g., ``be stressed out'' precedes ``relieve stress''). We use distributed event representation based on the Role Factored Tensor Model for a robust matching of event causality relations due to limited event causality knowledge of the system. Experimental results showed that the proposed method improved coherency and dialogue continuity of system responses.
BibTeX
@inproceedings{tanaka2019conversational,
  author = {Shohei Tanaka and Koichiro Yoshino and Katsuhito Sudoh and Satoshi Nakamura},
  title = {{Conversational Response Re-ranking Based on Event Causality and Role Factored Tensor Event Embedding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure