Abstract
In this work, we study how the finetuning stage in the pretrain-finetune framework changes the behavior of a pretrained neural language generator. We focus on the transformer encoder-decoder model for the open-domain dialogue response generation task. Our major finding is that after standard finetuning, the model forgets some of the important language generation skills acquired during large-scale pretraining. We demonstrate the forgetting phenomenon through a set of detailed behavior analysis from the perspectives of knowledge transfer, context sensitivity, and function space projection. As a preliminary attempt to alleviate the forgetting problem, we propose an intuitive finetuning strategy named "mix-review". We find that mix-review effectively regularizes the finetuning process, and the forgetting problem is alleviated to some extent. Finally, we discuss interesting behavior of the resulting dialogue model and its implications.
Introduction
The "pretrain-finetune" paradigm has become the backbone of modern Natural Language Processing (NLP). Models are first trained on massive, diverse corpora (e.g., CCNEWS) to acquire broad linguistic knowledge and then "specialized" on smaller, task-specific datasets like dialogue logs. However, this process often leads to "catastrophic forgetting." In the context of open-domain dialogue, models frequently lose the world knowledge and context sensitivity they gained during pretraining, resulting in generic, uninformative responses.
As shown in Figure 1, during standard finetuning, while the training loss on the target dialogue data decreases, the Negative Log-Likelihood (NLL) on the original pretraining validation set rises sharply. This indicates the model is actively losing its ability to model the broader language distribution it was initially taught.
Methodology: The Mix-Review Strategy
To address this, we propose Mix-review (MR). The core idea is simple: instead of finetuning exclusively on the target dataset, we re-introduce a subset of the original pretraining data during each epoch of finetuning.
The strategy involves two key hyperparameters:
- Mix-ratio: The ratio of pretraining data to target data included in each batch. A ratio of 4 was found to be effective in our experiments.
- Mix-decay: A factor that gradually reduces the proportion of pretraining data over time, allowing the model to focus more on the target task while maintaining a "memory" of its prior knowledge.
We compared Mix-review against standard $L_2$ regularization and weight decay toward the pretrained parameters ($WD(\theta_{pre})$), finding that Mix-review provides a more direct and effective regularization of the model's behavior.
Analysis: Parameter Space vs. Function Space
A critical insight of this work is the distinction between how finetuning affects a model's weights (parameter space) versus its output distributions (function space). Using UMAP projections, we visualized the trajectories of models under different finetuning regimes.
While models stay relatively close to their pretrained starting point in parameter space, their behavior in function space drifts significantly (Figure 6). This explains why simple weight-based regularization often fails to prevent forgetting; small shifts in the high-dimensional weight space can lead to massive changes in the model's actual responses. Mix-review effectively constrains the model's trajectory in function space, preserving context sensitivity and world knowledge.
Experimental Results
We evaluated Mix-review across several dialogue datasets, including Dailydialog, Switchboard, and Cornell Movie.
- Perplexity (PPL): Mix-review consistently achieved the lowest perplexity. On Dailydialog, the MR model reached a PPL of 11.07, significantly outperforming models trained from scratch (24.83) or standard finetuned models.
- Knowledge Retention: Using a benchmark derived from Google Trends 2016, we measured how well models retained factual information. The Mix-review model achieved a BLEU-2 score of 0.261 on Dailydialog, compared to 0.162 for standard finetuning.
- Human Evaluation: In multi-turn "knowledgeable" chats, human evaluators rated the Mix-review model higher across several dimensions, including Knowledge (3.40 vs 2.82), Consistency, and Engagingness.
Conclusion
Standard finetuning often acts as a "de-sensitization" process, stripping pretrained models of their broad capabilities. Mix-review serves as a powerful regularizer that preserves these skills by continuously reminding the model of its "education" during the specialization phase. While challenges like "boring" responses remain, alleviating forgetting is a crucial step toward more intelligent and responsive dialogue systems.
Summary updated