TL;DR
This paper explores a novel training objective for generative dialogue models that incorporates a "semantic loss" alongside the standard negative log-likelihood (NLL). The goal is to encourage more diverse and semantically relevant responses, especially in data-scarce scenarios. They found that this semantic loss can indeed improve response diversity on smaller datasets, and surprisingly, that using pre-trained language model embeddings (like BERT) as the semantic loss is more effective than using them for initial model weights.
What Problem Does This Paper Address?
Generative dialogue models, when trained with traditional objectives like NLL or cross-entropy, tend to mimic the exact sequences of words seen in the training data. This often leads to models that produce repetitive or overly safe responses, lacking the creativity and semantic variation that makes human conversation engaging. While there's a desire for models to generate diverse and semantically similar alternatives to a given context, existing training methods don't explicitly encourage this. This is particularly problematic in goal-oriented dialogue systems where collecting large, high-quality datasets is expensive, forcing models to learn from limited examples. The paper aims to bridge this gap by introducing and evaluating a semantic loss that pushes models to generate responses that are not just syntactically correct but also semantically aligned with the target, thereby fostering diversity.
Methods
The core of the paper's methodology is the introduction of a "semantic loss" ($L_{SEM}$) as an auxiliary objective during training. This loss is computed in three steps:
- Sampling: The model generates a response by sampling tokens from its distribution.
- Embedding Averaging: The word embeddings of the sampled response and the ground truth response are averaged. These embeddings are derived from pre-trained language models like BERT, GloVe, or fastText.
- L2 Distance: The L2 distance between the averaged embeddings of the sampled and ground truth responses is calculated. This distance serves as the semantic loss.
This semantic loss is then combined with the standard NLL loss ($L_{MLE}$) into a combined training objective ($L_{Train} = L_{MLE} + \alpha \cdot L_{SEM}$), where $\alpha$ is a hyperparameter controlling the strength of the semantic loss. To handle the non-differentiable sampling step, the REINFORCE algorithm is employed. The authors also investigate the effectiveness of using BERT embeddings for initializing the model weights versus using them within the semantic loss function.
Datasets
The paper uses two datasets for training and evaluation:
- Frames: A smaller dataset for task-oriented dialogue systems.
- MultiWoZ 2.0: A larger, multi-domain dataset for task-oriented dialogue.
Key Insights and Ideas
- Semantic Loss for Diversity: The semantic loss, when applied to smaller datasets, demonstrably improves response diversity. This suggests that explicitly rewarding semantic similarity encourages models to explore a wider range of valid responses.
- BERT Embeddings Shine in Loss, Not Initialization: Pre-trained language model embeddings (specifically BERT) are more effective when used as part of the semantic loss function rather than solely for initializing the model's word embeddings. This implies that the guidance provided by semantic similarity during training is more impactful than just providing a good starting point.
- Data Size Matters: The effectiveness of the semantic loss is highly dependent on the dataset size. While it significantly boosts diversity in smaller datasets like Frames, its impact is less pronounced in larger datasets like MultiWoZ, where the inherent diversity of the training data might already be sufficient.
- BLEU vs. Diversity: The study highlights a common challenge: metrics like BLEU, which measure overlap with reference responses, can decrease when diversity increases. This underscores the need for evaluation metrics that capture both fluency and novelty.
Key Results
- Frames Dataset: Minimizing the semantic loss improved response diversity (measured by distinct-1 and distinct-2 n-grams) on the smaller Frames dataset. Human evaluations also showed that models trained with the semantic loss were perceived as more diverse.
- MultiWoZ Dataset: The semantic loss did not yield significant improvements in diversity on the larger MultiWoZ dataset compared to the baseline NLL training.
- BERT Initialization vs. Semantic Loss: Models trained with BERT embeddings in the semantic loss ($Init1;Loss2$) showed greater diversity improvements than those using BERT for initialization ($Init2;Loss1$).
- % Unseen Bigrams: The semantic loss helped foster the generation of new, unseen bigrams, indicating a move away from simply mimicking training data patterns.
Limitations and Future Work
- Large Dataset Performance: The semantic loss's limited effectiveness on large datasets like MultiWoZ is a key limitation. The authors conjecture this is due to the higher frequency of n-grams in larger datasets, making it harder for the semantic loss to break the NLL bias.
- Out-of-NLL Sampling Challenges: Attempts to further break rigid sampling distributions by dropping tokens led to increased diversity but also introduced significant noise, making responses illegible.
- Future Work: The authors suggest exploring ways to make the random exploration (like token dropping) more robust by adding constraints to prevent divergence. They also note that the semantic loss was effective in low-data settings, implying its potential for few-shot learning scenarios.
Summary updated