Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models

Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, Fuchun Peng

Abstract

In this work, we study how the finetuning stage in the pretrain-finetune framework changes the behavior of a pretrained neural language generator. We focus on the transformer encoder-decoder model for the open-domain dialogue response generation task. Our major finding is that after standard finetuning, the model forgets some of the important language generation skills acquired during large-scale pretraining. We demonstrate the forgetting phenomenon through a set of detailed behavior analysis from the perspectives of knowledge transfer, context sensitivity, and function space projection. As a preliminary attempt to alleviate the forgetting problem, we propose an intuitive finetuning strategy named "mix-review". We find that mix-review effectively regularizes the finetuning process, and the forgetting problem is alleviated to some extent. Finally, we discuss interesting behavior of the resulting dialogue model and its implications.

Introduction

The modern NLP landscape is dominated by the "pretrain then finetune" paradigm. Models are first trained on massive, diverse corpora to acquire general linguistic knowledge and then specialized on smaller, task-specific datasets. While this approach has yielded state-of-the-art results across various benchmarks, it introduces a significant side effect: the loss of general intelligence in favor of task-specific performance. This paper investigates this "forgetting" phenomenon specifically within the context of open-domain dialogue response generation.

The Forgetting Phenomenon

Open-domain dialogue is uniquely vulnerable to forgetting. To be an effective conversationalist, a model requires both "world knowledge" (factual depth) and "context sensitivity" (the ability to track conversation history). However, dialogue datasets are often "thin," consisting of generic banter that lacks the factual richness of pretraining data (e.g., news or Wikipedia).

Standard finetuning often acts as a "brain dump," where the model discards its broad knowledge to over-fit the narrow stylistic patterns of the chat data. As shown in Figure 1, while the model's performance on the target dialogue task improves (decreasing NLL), its performance on the original pretraining validation set (CCNEWS) degrades rapidly, indicating a loss of previously acquired skills.

Figure 1: Visualization of the forgetting phenomenon during standard finetuning, where NLL on pretraining data rises as target data is learned.

Methodology: The Mix-review Strategy

To combat this forgetting, the authors propose Mix-review (MR). This strategy involves mixing a random subset of the original pretraining data into the finetuning process. This acts as a form of rehearsal, reminding the model of its original "wisdom" while it learns the new task. Two key hyperparameters govern this process:

  1. Mix-ratio: The proportion of pretraining data included relative to the target dialogue data.
  2. Mix-decay: A factor that gradually reduces the amount of pretraining data as finetuning progresses, allowing the model to eventually specialize more on the dialogue task.

Compared to standard regularization techniques like weight decay ($L_2$ regularization), Mix-review is significantly more effective at stabilizing the model's performance on pretraining data without sacrificing gains on the target task.

Figure 2: Comparison between Mix-review and standard Weight Decay (WD). Mix-review (a) successfully stabilizes the pretraining validation loss compared to WD (b).

Behavioral and Structural Analysis

The researchers utilized UMAP projections to visualize changes in the model's parameter space (raw weights) versus its function space (output distributions). A key insight was that while the model's parameters changed relatively little during finetuning, its actual behavior shifted drastically. This suggests that even small weight updates in a high-dimensional space can lead to massive forgetting of complex linguistic patterns.

Furthermore, standard finetuning was found to induce "Generic Response Syndrome." Models became less sensitive to context, falling back on safe but uninformative responses (e.g., "I don't know") because they lost the ability to utilize the complex linguistic structures learned during pretraining.

Key Results and Evaluation

The effectiveness of Mix-review was validated through several metrics across multiple datasets (Dailydialog, Switchboard, Cornell Movie):

  • Perplexity Gains: Mix-review consistently achieved the lowest test-set perplexity. On Dailydialog, the MR model reached a perplexity of 11.07, a stark improvement over the 24.83 achieved by a model trained from scratch.
  • Knowledge Retention: When prompted with specific terms from Google Trends 2016 (e.g., "Pokemon Go"), the MR model could still generate factual descriptions, whereas standard finetuned models provided generic, vague answers.
  • Human Evaluation: In multi-turn evaluations, human judges rated the Mix-review model significantly higher for Knowledge, Consistency, and Engagement.

Conclusion

While the pretrain-finetune framework is powerful, standard finetuning risks erasing the very "intelligence" that makes large-scale pretraining valuable. Mix-review offers a simple, intuitive path toward creating dialogue models that are both task-proficient and broadly knowledgeable. Future work remains to address the persistent issue of "boring" responses and the increased sensitivity of these models to malicious provocation.

Summary updated

Abstract

In this work, we study how the finetuning stage in the pretrain-finetune framework changes the behavior of a pretrained neural language generator. We focus on the transformer encoder-decoder model for the open-domain dialogue response generation task. Our major finding is that after standard finetuning, the model forgets some of the important language generation skills acquired during large-scale pretraining. We demonstrate the forgetting phenomenon through a set of detailed behavior analysis from the perspectives of knowledge transfer, context sensitivity, and function space projection. As a preliminary attempt to alleviate the forgetting problem, we propose an intuitive finetuning strategy named "mix-review". We find that mix-review effectively regularizes the finetuning process, and the forgetting problem is alleviated to some extent. Finally, we discuss interesting behavior of the resulting dialogue model and its implications.
BibTeX
@misc{he2019analyzing,
  author = {Tianxing He and Jun Liu and Kyunghyun Cho and Myle Ott and Bing Liu and James Glass and Fuchun Peng},
  title = {{Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models}},
  year = {2019},
  eprint = {1910.07117},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.07117},
}
Figure