TL;DR
This paper introduces DECODE, a new dataset and task for detecting contradictions in dialogues. They propose a structured utterance-based approach that significantly outperforms standard methods, especially in out-of-domain scenarios, and show its utility in evaluating and improving chatbot consistency.
What Problem Does This Paper Address?
Building truly human-like open-domain chatbots is a significant challenge. One major hurdle is ensuring conversational consistency, as chatbots often produce contradictory statements. These inconsistencies not only disrupt the flow of conversation but also erode user trust and confidence. While previous work has explored dialogue consistency, there's a lack of effective methods to measure and address contradictions, especially in real-world, out-of-domain human-bot interactions. This paper aims to fill this gap by introducing a dedicated task and dataset for dialogue contradiction detection and proposing a more robust modeling approach.
Methods
The authors introduce the DialoguE COntradiction DEtection (DECODE) task, which formalizes contradiction detection as a supervised classification problem. The goal is to identify if the last utterance in a dialogue contradicts any previous information.
They propose two main modeling approaches:
- Unstructured Approach: This is a standard approach where the entire dialogue history is concatenated into a single text sequence and fed into a Transformer-based Natural Language Inference (NLI) model. Special tokens are used to indicate speaker turns.
- Structured Utterance-based Approach: This novel approach explicitly leverages the dialogue structure. It pairs each utterance by the last speaker with the last utterance and feeds these pairs into a Transformer NLI model. The final contradiction probability is the maximum over these pairs. This method also allows for identifying supporting evidence for the contradiction.
Datasets
The paper introduces the DECODE dataset, which consists of:
- Main DECODE Dataset: 17,713 human-written dialogues with deliberate contradictions, collected from existing corpora (Wizard of Wikipedia, EMPATHETIC DIALOGUES, Blended Skill Talk, ConvAI2). Annotators were asked to continue dialogues such that the last utterance contradicted previous ones and to mark the evidence.
- Auxiliary Test Sets:
- Add Two Turns (A2T): Contradictory dialogues with randomly sampled turns inserted to increase dialogue history length.
- Remove Contradicting Turns (RCT): Non-contradictory dialogues created by removing the turns marked as evidence for the contradiction.
- Human-Bot Test Set: Dialogues collected from crowdworkers interacting with various open-domain bots, where contradictions made by the bots were marked. This set is crucial for evaluating out-of-domain performance.
Existing NLI datasets like SNLI, MNLI, DNLI, and ANLI-R3 were also used for training and comparison.
Key Insights and Ideas
- DECODE is a superior dataset: The authors found that their newly collected DECODE dataset provides significantly better supervision for dialogue contradiction detection compared to existing NLI datasets, even those specifically designed for dialogue.
- Structure matters: Explicitly modeling dialogue structure (the utterance-based approach) leads to more robust models that generalize better, especially to out-of-domain human-bot conversations. This challenges the assumption that standard Transformers can implicitly learn all necessary structure.
- Human-bot consistency is a distinct challenge: Models that perform well on human-human dialogues can struggle with human-bot interactions, highlighting the need for specialized evaluation and training data.
- Human-like consistency is still a gap: Despite advancements, there's a significant gap between the performance of state-of-the-art models and human judgment in identifying contradictions.
Key Results
- Improved Accuracy: The utterance-based approach trained on DECODE achieved significantly higher accuracy on the DECODE Main test set (93.19%) compared to models trained on other NLI data (e.g., ANLI-R3: 82.21%, DNLI: 76.54%).
- Out-of-Domain Robustness: The utterance-based approach showed much better transferability to the Human-Bot test set (84.69% accuracy) compared to the unstructured approach (70.03% accuracy).
- Evidence Retrieval: The structured approach also improved the F1 score for supporting evidence retrieval.
- Correlation with Human Judgment: The best contradiction detection model correlated well with human judgments (Pearson correlation of 0.81), suggesting its potential as an automatic metric.
- Generation Improvement: Using the DECODE detector for re-ranking chatbot generations (specifically with Top-k sampling) significantly reduced the contradiction rate, both by the automatic metric and human evaluation.
Limitations and Future Work
- Human-Bot Gap: The paper acknowledges a significant gap between the best automatic detectors and human judgment in identifying contradictions, indicating that there's still much room for improvement.
- Pre-training Data: The authors speculate that models pre-trained on broader dialogue-focused corpora might perform even better, suggesting future work in this area.
- Generalizability of Unstructured Models: The unstructured approach was found to overfit on superficial patterns in the training data, leading to poor performance on out-of-domain sets. This highlights the importance of inductive biases for generalization.
- Future Work: The authors suggest exploring more sophisticated methods for dialogue consistency modeling and further investigating the use of contradiction detectors for improving chatbot generation. They also emphasize the need for continued research to close the gap between model and human performance.
Summary updated