A Brief Study on the Effects of Training Generative Dialogue Models with a Semantic loss

Prasanna Parthasarathi, Mohamed Abdelsalam, Joelle Pineau, Sarath Chandar

TL;DR

Standard dialogue models trained with negative log-likelihood (NLL) tend to mimic exact training phrases, leading to repetitive and generic responses. This paper introduces an auxiliary semantic loss objective using pretrained language model embeddings to reward responses that are semantically similar to the target, even if they use different words. This approach significantly boosts response diversity in low-data scenarios without sacrificing relevance.

What Problem Does This Paper Address?

When training generative dialogue models, the standard go-to objective is Maximum Likelihood Estimation (MLE) via Negative Log-Likelihood (NLL) or Cross-Entropy loss. These metrics evaluate success at the token level.

Consider this example from the paper:

  • Target: "Would you like to travel to Paris?"
  • Response 1 (R1): "How about Paris as your destination?"
  • Response 2 (R2): "Would she like to read to me?"

To a human, R1 is a perfectly valid, semantically equivalent alternative. R2 is complete nonsense. However, because R2 shares four exact tokens in the same positions as the target, a standard NLL loss will actually favor R2 over R1.

This token-level bias forces models to rigidly memorize exact n-gram patterns from the training data. The problem is severely amplified in small or niche datasets, where the lack of diverse examples causes the model to generate safe, repetitive, and uninformative responses.

Methods

The authors propose a training objective ($L_{Train}$) that combines standard MLE loss with an auxiliary Semantic Loss ($L_{SEM}$):

$$L_{Train} = L_{MLE} + \alpha * L_{SEM}$$

The semantic loss is calculated in three distinct steps during training:

  1. Sampling: Instead of greedy decoding, the model samples a response sequence from the decoder's probability distribution.
  2. Embedding & Distance: The tokens of both the sampled response and the ground-truth target response are converted into dense vectors using a pretrained language model (such as BERT, GloVe, or fastText). These token vectors are averaged to create a single sentence embedding for each, and the $L_2$ distance between the two sentence embeddings is calculated.
  3. Optimization via Reinforcement Learning: Because the token sampling step is non-differentiable, the authors use the REINFORCE algorithm to backpropagate the gradient, treating the negative semantic distance as a reward.

Datasets

The approach was evaluated on two goal-oriented dialogue datasets of contrasting sizes:

  • Frames: A relatively small dataset (30k unique context bigrams, 127k total target bigrams) focused on travel booking.
  • MultiWoZ 2.0: A much larger, multi-domain wizard-of-oz dataset (40k unique context bigrams, 900k total target bigrams).

Key Insights and Ideas

  • The Dataset Size Paradox: The semantic loss worked beautifully on the smaller Frames dataset but showed a "negative result" on the larger MultiWoZ dataset. In large datasets, frequent bigrams appear so often that the model develops a massive sampling bias toward them, making out-of-distribution exploration incredibly difficult.
  • Loss Function vs. Initialization: Surprisingly, using BERT embeddings to calculate the semantic loss yielded much higher response diversity than simply using BERT to initialize the model's word embeddings. Initializing with rigid pretrained embeddings can actually lock the model into a local minimum, stifling exploration.

Key Results

The quantitative effects of the semantic loss across different embedding types can be seen in the training curves below:

Figure 1: Quantitative comparison of different loss functions, and initialization on Frames and MultiWoZ data sets.
  • The BLEU vs. Diversity Trade-off: On the Frames dataset, models trained with semantic loss ($L_{Train}$) showed a drop in BLEU scores compared to the baseline (Figure 1a). However, this was accompanied by a massive spike in Distinct-2 (unique bigrams) and % Unseen Bigrams (Figures 1b and 1c). The model was actively learning to express the same ideas using entirely new vocabulary.
  • Human Evaluation Validation: Because automated metrics like BLEU penalize vocabulary changes, the authors conducted human studies. Human scorers overwhelmingly preferred the semantic loss model for diversity, choosing its responses as more diverse/interesting 63% to 65% of the time, while maintaining comparable contextual relevance to the baseline.

Limitations and Future Work

While effective for small datasets, the proposed semantic loss struggled to break the rigid sampling distributions inherent to large datasets like MultiWoZ. When the authors tried to force exploration in MultiWoZ by randomly masking/dropping tokens during decoding, the training diverged, resulting in ungrammatical and noisy text.

The authors note that a key direction for future work is introducing additional syntactic constraints to ensure that when the model explores semantically diverse spaces, it still maintains grammatical structure.

Summary updated

Abstract

Neural models trained for next utterance generation in dialogue task learn to mimic the n-gram sequences in the training set with training objectives like negative log-likelihood (NLL) or cross-entropy. Such commonly used training objectives do not foster generating alternate responses to a context. But, the effects of minimizing an alternate training objective that fosters a model to generate alternate response and score it on semantic similarity has not been well studied. We hypothesize that a language generation model can improve on its diversity by learning to generate alternate text during training and minimizing a semantic loss as an auxiliary objective. We explore this idea on two different sized data sets on the task of next utterance generation in goal oriented dialogues. We make two observations (1) minimizing a semantic objective improved diversity in responses in the smaller data set (Frames) but only as-good-as minimizing the NLL in the larger data set (MultiWoZ) (2) large language model embeddings can be more useful as a semantic loss objective than as initialization for token embeddings.
BibTeX
@inproceedings{parthasarathi2021brief,
  author = {Prasanna Parthasarathi and Mohamed Abdelsalam and Joelle Pineau and Sarath Chandar},
  title = {{A Brief Study on the Effects of Training Generative Dialogue Models with a Semantic loss}},
  booktitle = {SIGDIAL},
  year = {2021},
  doi = {10.18653/v1/2021.sigdial-1.49},
}
Figure