TL;DR
This paper introduces LSTDial, a novel two-stage framework for dialogue generation that leverages both turn-level and dialogue-level quality feedback during training. By explicitly incorporating these feedback signals, LSTDial significantly improves response quality, leading to more coherent, engaging, and informative conversations compared to traditional methods.
What Problem Does This Paper Address?
Generating high-quality responses in open-domain dialogue systems remains a significant challenge. While various quality dimensions like coherence, diversity, and relevance exist for evaluating dialogues, current systems primarily rely on optimizing the average likelihood of training data (Maximum Likelihood Estimation, MLE). This approach often overlooks explicit quality signals, creating a gap between generation and evaluation. Consequently, dialogue systems struggle to consistently produce responses that are not only fluent and relevant at a single turn but also maintain coherence and engagement throughout an entire conversation.
Methods
LSTDial proposes a two-stage training framework:
- Stage 1: Short-term Training (STDial): This stage focuses on improving turn-level response quality. It involves fine-tuning a pre-trained dialogue system using a multi-objective loss function that combines MLE with short-term feedback. This feedback is generated by a Turn-level Evaluator (TEval), which assesses qualities like fluency, relevance, and diversity for individual turns.
- Stage 2: Long-term Training (LTDial): This stage builds upon the first by enhancing dialogue-level quality using Reinforcement Learning (RL). It trains the system further by incorporating long-term feedback as a reward signal. This feedback is generated by a Dialogue-level Evaluator (DEval), which considers engagement, coherence, and informativeness across the entire conversation. To estimate future rewards, a user simulator is employed to generate hypothetical follow-up turns.
The novelty lies in the explicit integration of these multi-faceted quality evaluations as feedback mechanisms within a structured training process, bridging the gap between generation and evaluation.
Datasets
The paper uses two English open-domain dialogue datasets:
- DailyDialog: A manually labeled multi-turn dialogue dataset.
- Opensubtitles: A large parallel corpus extracted from movie and TV subtitles.
The datasets were pre-processed to extract single-turn dialogues for training and evaluation.
Key Insights and Ideas
- Bridging the Generation-Evaluation Gap: The core idea is that explicitly training a dialogue system with feedback that mirrors human evaluation criteria (both at the turn and dialogue level) leads to demonstrably better conversational agents.
- Synergy of Short- and Long-Term Feedback: The two-stage approach is crucial. Short-term feedback hones individual responses, while long-term feedback guides the system to maintain conversational flow and engagement over multiple turns. Removing either stage significantly degrades performance.
- RL for Long-Term Coherence: Reinforcement Learning proves effective for optimizing dialogue-level objectives like coherence and engagement, which are harder to capture with standard MLE training.
- Learned Evaluators are Key: The success hinges on the quality of the learned evaluators (TEval and DEval) that can reliably predict human judgments on various quality dimensions.
Key Results
- Automatic Evaluation: LSTDial consistently outperformed baseline models (Transformer, GPT-2, PLATO, BART) across various automatic metrics (Dist-1, Dist-2, Dist-3, BLEU-1 to BLEU-4, PPL) on both DailyDialog and OpenSubtitles datasets. For instance, on DailyDialog, LSTDial achieved a significant improvement in Dist-3 (32.14 vs. 15.13 for the baseline Transformer).
- Human Evaluation: Human evaluation on the DailyDialog dataset showed that LSTDial significantly improved turn-level qualities (Fluency, Relevance, Diversity) and dialogue-level qualities (Engagement, Coherence, Informativeness) compared to baseline models.
- LLM Applicability: The framework was successfully adapted to a small LLM (InstructGPT), demonstrating its effectiveness and showing slight improvements over the base LLM.
- Qualitative Improvements: Case studies revealed that LSTDial generates more diverse, interesting, and engaging responses, often asking follow-up questions and expressing emotions, leading to deeper conversations.
Limitations and Future Work
- Resource Constraints: The authors acknowledge that they did not use larger evaluation models or LLMs with more parameters due to resource limitations.
- Initialization Strategy: The RL iteration phase defaulted to using the first utterance as the initial dialogue history, which could be further optimized by using more initial utterances for richer context.
- Training Order Dependency: The LSTDial framework requires a specific training order (short-term first, then long-term) for optimal convergence.
- Future Work: The authors suggest exploring more sophisticated feedback mechanisms and further leveraging LLMs for dialogue evaluation to enhance generation quality. They also plan to investigate the impact of different initialization strategies for the RL stage.
Summary updated