Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Tasks

Yufan Zhao, Can Xu, Wei Wu

Abstract

We study multi-turn response generation for open-domain dialogues. The existing state-of-the-art addresses the problem with deep neural architectures. While these models improved response quality, their complexity also hinders the application of the models in real systems. In this work, we pursue a model that has a simple structure yet can effectively leverage conversation contexts for response generation. To this end, we propose four auxiliary tasks including word order recovery, utterance order recovery, masked word recovery, and masked utterance recovery, and optimize the objectives of these tasks together with maximizing the likelihood of generation. By this means, the auxiliary tasks that relate to context understanding can guide the learning of the generation model to achieve a better local optimum. Empirical studies with three benchmarks indicate that our model can significantly outperform state-of-the-art generation models in terms of response quality on both automatic evaluation and human judgment, and at the same time enjoys a much faster decoding process.

Introduction

In the field of open-domain conversational AI, generating relevant responses in a multi-turn dialogue remains a significant challenge. To produce a coherent reply, a model must grasp the hierarchical structure of the conversation, where words form utterances and utterances form a broader context. Recent research has trended toward increasingly deep and complex neural architectures, such as ReCoSa, to capture these dependencies. However, these models are often parameter-heavy and slow to decode, making them difficult to deploy in latency-sensitive environments like mobile applications.

This paper explores whether the burden of context understanding can be shifted from the model's architectural complexity to the learning process itself. The authors demonstrate that a simple, single-layer Transformer model can outperform complex state-of-the-art models when trained with clever auxiliary tasks, achieving faster decoding speeds and a significantly smaller memory footprint.

Methodology

The proposed architecture is intentionally "frugal," consisting of a simple Transformer-based encoder-decoder with only one attentive layer in both the encoder and the decoder.

Overall architecture of the proposed model, showing the encoder-decoder structure and the integration of auxiliary tasks.

To empower this shallow model, the authors introduce four auxiliary self-supervised tasks that serve as regularization during training:

  1. Word Order Recovery (WOR): The model predicts the original positions of words from a shuffled utterance, forcing it to learn local sequential dependencies.
  2. Masked Word Recovery (MWR): Similar to BERT, the model predicts masked words (15% of the input) based on the surrounding context.
  3. Masked Utterance Recovery (MUR): An entire utterance is replaced with a [MASK] token, requiring the model to infer the missing part of the conversation.
  4. Utterance Order Recovery (UOR): The model reconstructs the correct sequence from shuffled utterances in a dialogue context, teaching it the logical flow of conversation.
The four auxiliary tasks used to guide the model's learning: (a) Word Order Recovery, (b) Masked Word Recovery, (c) Masked Utterance Recovery, and (d) Utterance Order Recovery.

The final training objective is a joint loss function combining standard Maximum Likelihood Estimation (MLE) for response generation with the weighted losses of these four auxiliary tasks.

Experiments and Results

The model was evaluated on three diverse benchmarks: DailyDialog (daily life topics), PERSONA-CHAT (persona-based chit-chat), and the Ubuntu Dialogue Corpus (technical support logs).

Key Findings

  • Learning vs. Modeling: Context understanding does not strictly require deep layers if the training signal is sufficiently rich. Auxiliary tasks guide the simple model toward a superior local optimum that it wouldn't find with standard MLE alone.
  • Depth-Equivalency: Ablation studies revealed that a 1-layer model trained with auxiliary tasks performs as well as much deeper models (up to 7 layers) trained with standard MLE.
  • Structural Importance: On DailyDialog and Ubuntu, the order recovery tasks (WOR and UOR) were more critical than masking tasks, suggesting that understanding conversational structure is vital for response generation.
  • Efficiency: The shallow architecture avoids the "complexity tax" of deep Transformers. The model is 2x faster at decoding than ReCoSa and uses significantly fewer parameters (e.g., 14.4M vs. 73.8M on DailyDialog).

Performance

The proposed model consistently outperformed state-of-the-art baselines like ReCoSa and HRAN in terms of Perplexity (PPL), BLEU scores, and response diversity (Distinct-1/2). Human evaluations also favored the model for its informativeness, consistency, and fluency.

Conclusion

This work demonstrates that simple models can be highly effective for multi-turn response generation when paired with well-designed auxiliary tasks. While there are diminishing returns as model depth increases, the "simple model + complex tasks" approach offers a powerful path toward efficient and high-quality dialogue systems. Future research may explore applying this philosophy to other natural language generation domains beyond dialogue.

Summary updated

Abstract

We study multi-turn response generation for open-domain dialogues. The existing state-of-the-art addresses the problem with deep neural architectures. While these models improved response quality, their complexity also hinders the application of the models in real systems. In this work, we pursue a model that has a simple structure yet can effectively leverage conversation contexts for response generation. To this end, we propose four auxiliary tasks including word order recovery, utterance order recovery, masked word recovery, and masked utterance recovery, and optimize the objectives of these tasks together with maximizing the likelihood of generation. By this means, the auxiliary tasks that relate to context understanding can guide the learning of the generation model to achieve a better local optimum. Empirical studies with three benchmarks indicate that our model can significantly outperform state-of-the-art generation models in terms of response quality on both automatic evaluation and human judgment, and at the same time enjoys a much faster decoding process.
BibTeX
@inproceedings{zhao2020learning,
  author = {Yufan Zhao and Can Xu and Wei Wu},
  title = {{Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Tasks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=RYDBaxn-2vh},
}

Loading related content...

Figure