Cross Copy Network for Dialogue Generation

Changzhen Ji, Xin Zhou, Yating Zhang, Xiaozhong Liu, Changlong Sun, Conghui Zhu, Tiejun Zhao

Introduction

Standard dialogue models frequently encounter difficulties when dealing with domain-specific logic and out-of-vocabulary (OOV) terms. While Pointer-Generator Networks (PGN) mitigate this by enabling models to copy words from the immediate dialogue context, they remain restricted to the current session's information. In specialized fields like law or customer service, human experts often rely on precedents—recurring patterns from historical data. The Cross Copy Network (CCN) addresses this by enabling "horizontal" retrieval across datasets to guide response generation.

Conceptual Framework: Vertical and Horizontal Copying

The core innovation of CCN lies in its dual-source copying mechanism. "Vertical copy" refers to extracting information from the current conversation's history, a standard practice in PGNs. In contrast, "horizontal copy" allows the model to reference similar historical cases retrieved from the training set. This allows the model to "borrow" complex phrasing, such as legal citations or specific product specifications, that it might otherwise fail to generate correctly.

Conceptual illustration of vertical and horizontal copying in legal dialogues.

The Cross Copy Network (CCN) Architecture

The CCN architecture is designed to integrate these two information streams through a multi-stage process:

  1. Target Case Representation: The current dialogue is encoded using a hierarchical structure consisting of utterance-level Bi-LSTMs, a dialogue-level attention layer, and a Transformer layer to capture immediate context.
  2. Similar Case Retrieval: Using ElasticSearch, the system identifies the top 50 candidate dialogues from the training data. These are subsequently re-ranked using a fine-tuned RoBERTa model to select the most relevant "Similar Cases" (SC).
  3. Cross Copy Decoder: The decoder computes two distinct pointer distributions: $\alpha$ for vertical copying (current context) and $\beta$ for horizontal copying (retrieved cases). The final output is a mixture of the standard vocabulary distribution and these two copy mechanisms.
The Cross Copy Decoder architecture, illustrating the integration of target and similar case encoders with the vocabulary distribution.

Experimental Evaluation

The model was rigorously tested on two large-scale datasets: the Court Debate Dataset (CDD), containing 121,016 private lending disputes, and the Jing Dong Dialogue Corpus (JDDC), featuring over 1 million e-commerce customer service interactions.

Key Results

CCN achieved state-of-the-art performance across both quantitative and qualitative metrics. On the CDD dataset, the CCN-top3 configuration reached a BLEU score of 24.75, significantly outperforming the standard Transformer (18.58) and PGN (18.75). Human evaluation further confirmed these findings, with CCN outputs receiving higher scores for relevance and fluency. Notably, the model demonstrated a superior ability to replicate complex legal logic by successfully "borrowing" phrasing from historical precedents.

Discussion and Future Work

While CCN represents a significant advancement, error analysis indicates that approximately 50% of failures occur when required information is absent from both the current context and retrieved cases. This suggests that future iterations should incorporate broader external knowledge bases, such as legal textbooks or product manuals. Additionally, addressing the computational overhead of the retrieval step and improving the coherence of long-form responses remain key areas for development.

Conclusion

The Cross Copy Network demonstrates that dialogue generation in specialized domains can be effectively framed as a retrieval-augmented generation problem. By formalizing the distinction between vertical and horizontal copying, CCN provides a robust framework for leveraging historical data to produce more accurate and logically sound dialogue.

Summary updated

Abstract

In the past few years, audiences from different fields witness the achievements of sequence-to-sequence models (e.g., LSTM+attention, Pointer Generator Networks, and Transformer) to enhance dialogue content generation. While content fluency and accuracy often serve as the major indicators for model training, dialogue logics, carrying critical information for some particular domains, are often ignored. Take customer service and court debate dialogue as examples, compatible logics can be observed across different dialogue instances, and this information can provide vital evidence for utterance generation. In this paper, we propose a novel network architecture - Cross Copy Networks(CCN) to explore the current dialog context and similar dialogue instances' logical structure simultaneously. Experiments with two tasks, court debate and customer service content generation, proved that the proposed algorithm is superior to existing state-of-art content generation models.
BibTeX
@inproceedings{ji2020cross,
  author = {Changzhen Ji and Xin Zhou and Yating Zhang and Xiaozhong Liu and Changlong Sun and Conghui Zhu and Tiejun Zhao},
  title = {{Cross Copy Network for Dialogue Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.149},
}
Figure