Another Diversity-Promoting Objective Function for Neural Dialogue Generation

Ryo Nakamura, Katsuhito Sudoh, Koichiro Yoshino, Satoshi NakamuraarXiv 2018 · arXiv

TL;DR

Standard neural dialogue models often default to boring, generic responses like "I don't know" because traditional loss functions favor highly frequent words. This paper introduces Inverse Token Frequency (ITF) loss, a simple yet highly effective loss function that dynamically downweights frequent words and upweights rare ones during training. Without adding architectural complexity or training instability, ITF loss achieves state-of-the-art response diversity while maintaining high conversational quality.

What Problem Does This Paper Address?

If you've ever interacted with a basic chatbot, you've probably noticed it gets repetitive, frequently relying on safe, generic phrases like "I'm sorry" or "I don't know."

This low-diversity problem stems from how these models are trained. Standard sequence-to-sequence (Seq2Seq) models use Maximum Likelihood Estimation (MLE) paired with Softmax Cross-Entropy (SCE) loss. SCE treats every word class equally. Because generic phrases appear thousands of times in training data, the model learns that outputting them is a statistically "safe bet" to minimize loss.

While researchers have tried to fix this using complex frameworks like Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), or Reinforcement Learning (RL), these methods are notoriously unstable, difficult to implement, and often require extensive pre-training. The authors of this paper ask a simpler question: Can we fix the diversity problem by simply changing the loss function?

Methods

The authors propose Inverse Token Frequency (ITF) loss, a frequency-weighted modification of the standard Softmax Cross-Entropy loss.

Instead of treating every token equally, ITF loss scales the loss for each token based on how rarely it appears in the training set:

$$\mathcal{L}_{ITF} = w_c \mathcal{L}_{sce}$$

$$w_c = \frac{1}{\text{freq}(\text{token}_c)^\lambda}$$

Where:

  • $\text{freq}(\text{token}_c)$ is the frequency of the target token in the dataset.
  • $\lambda$ is a hyperparameter (set to $0.4$ in their experiments) that controls how aggressively the model penalizes frequent words.

By scaling down the loss for common words, the model is forced to focus on learning and generating rarer, more interesting vocabulary. Because this only changes the objective function, training remains just as stable as standard MLE.

Repetition Suppression

One side effect of encouraging rare tokens is that the model can occasionally get stuck in a loop repeating an unusual word. To counter this, the authors introduced a repetition suppressor into the decoder during inference, which dynamically lowers the probability of a token the more times it has already been generated in the current response.

Datasets

The authors evaluated their approach across different domains and languages using three large-scale datasets:

  1. English OpenSubtitles (2018 Corpus): A massive dataset of movie subtitles, processed into 4.6 million training examples.
  2. English Twitter Replies: 2.5 million multi-turn dialogue examples extracted from Twitter.
  3. Japanese Twitter Replies: 3.8 million multi-turn dialogue examples, tokenized using SentencePiece to handle Japanese characters effectively.

Key Insights and Ideas

  • Loss Functions > Complex Architectures: You don't need to bolt a complex GAN or VAE onto your model to get diverse outputs. Simply adjusting the gradients via the loss function can guide the model toward richer vocabulary.
  • Gradient Pacing: ITF loss essentially acts as a speed regulator for learning. It slows down learning on saturated, high-frequency tokens and accelerates learning on rare tokens, balancing out the natural zipfian distribution of human language.
  • Inference vs. Training Application: The authors also experimented with applying ITF during inference only (similar to Maximum Mutual Information reranking). While helpful, they found that training directly with ITF loss yielded the best balance of quality and diversity.

Key Results

The ITF loss model was evaluated using BLEU (to measure language quality/similarity to the ground truth) and DIST-1/2 (the percentage of distinct unigrams and bigrams, measuring diversity).

  • OpenSubtitles: The Seq2Seq + ITF model established a state-of-the-art DIST-1 score of 7.56, vastly outperforming standard Seq2Seq (1.43) and the popular MMI-antiLM method (6.54), while maintaining a competitive BLEU score.
  • Surpassing Human Baseline Diversity: On the Japanese Twitter dataset, the ITF loss model achieved a DIST-1 score of 16.8, which actually exceeded the diversity score of the human ground-truth text (16.2), effectively eliminating the "generic response" trap.

Qualitatively, as shown in the paper's examples, where a standard model responds to "I'm not interested in talking to you" with a generic "I'm going to the bathroom," the ITF model generates a much more natural, context-aware response: "I'll be back tomorrow morning."

Limitations and Future Work

While ITF loss drastically improves unigram (single-word) diversity, it operates purely at the token level. The authors note that future work will explore Inverse N-gram Frequency (INF) loss, which would scale losses based on the rarity of multi-word phrases rather than individual words. Additionally, because automated metrics like BLEU are notoriously imperfect for open-ended dialogue evaluation, the authors plan to validate these diversity gains with comprehensive human evaluations.

Summary updated

Abstract

Although generation-based dialogue systems have been widely researched, the response generations by most existing systems have very low diversities. The most likely reason for this problem is Maximum Likelihood Estimation (MLE) with Softmax Cross-Entropy (SCE) loss. MLE trains models to generate the most frequent responses from enormous generation candidates, although in actual dialogues there are various responses based on the context. In this paper, we propose a new objective function called Inverse Token Frequency (ITF) loss, which individually scales smaller loss for frequent token classes and larger loss for rare token classes. This function encourages the model to generate rare tokens rather than frequent tokens. It does not complicate the model and its training is stable because we only replace the objective function. On the OpenSubtitles dialogue dataset, our loss model establishes a state-of-the-art DIST-1 of 7.56, which is the unigram diversity score, while maintaining a good BLEU-1 score. On a Japanese Twitter replies dataset, our loss model achieves a DIST-1 score comparable to the ground truth.
BibTeX
@misc{nakamura2018another,
  author = {Ryo Nakamura and Katsuhito Sudoh and Koichiro Yoshino and Satoshi Nakamura},
  title = {{Another Diversity-Promoting Objective Function for Neural Dialogue Generation}},
  year = {2018},
  eprint = {1811.08100},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.08100},
}

Loading related content...

Figure