End-to-End Neural Context Reconstruction in Chinese Dialogue

Wei Yang, Rui Qiao, Haocheng Qin, Amy Sun, Luchen Tan, Kun Xiong, Ming LiACL 2019

TL;DR

This paper tackles the challenge of "context reconstruction" in Chinese dialogue, aiming to make conversational turns understandable in isolation by resolving pronouns and elided elements. They introduce a novel end-to-end neural network, POSNet, that jointly performs detection and resolution, achieving state-of-the-art results by leveraging novel data augmentation techniques.

What Problem Does This Paper Address?

Modern chatbots and dialogue systems are becoming increasingly sophisticated, but they often struggle with understanding conversations that span multiple turns. This is because human language, especially in dialogue, is rife with coreference (pronouns referring to entities) and ellipsis (omitted words or phrases). For instance, in a conversation, someone might say "I like it," where "it" refers to something mentioned in a previous turn. To process these turns independently for downstream tasks like question answering, we need to reconstruct the missing context. This paper addresses the problem of "context reconstruction" in Chinese dialogue, specifically focusing on resolving pronouns and zero pronouns (implicit references) to make each utterance semantically complete and independent.

Methods

The authors propose a novel end-to-end neural network architecture called POSNet. Their approach breaks down the context reconstruction problem into two main subtasks:

  • Detection: This subtask identifies referring expressions (like pronouns) and zero pronouns that need resolution. It's framed as a sequence labeling task. They employ POS (Part-of-Speech) tagging and positional embeddings, combined with CNNs and LSTMs, to encode the input sentence. A unique "pronoun masking mechanism" is also used.
  • Resolution: Once potential slots for resolution are identified, this subtask finds the most appropriate mention from the context to fill the slot. This is formulated as a ranking problem, where the model ranks candidate mentions.

The key innovation is the end-to-end joint training of these two modules. This allows the model to learn how detection errors might impact resolution and vice-versa, leading to improved overall performance. They also introduce a novel data augmentation strategy to generate a large amount of realistic training data, addressing the common issue of data scarcity in this domain.

Datasets

The paper utilizes several datasets:

  • CONLL2012: A standard dataset for coreference resolution, used for evaluating their zero pronoun resolution task.
  • OntoNote (BC/TC): Subsets of the OntoNote corpus were extracted and adapted for Chinese context reconstruction, specifically for zero pronoun resolution.
  • CQA (Chinese Question Answering): A large-scale dataset collected from Chinese CQA websites (Baidu Zhidao, Soso Wenwen). This dataset was specifically generated to address the need for large, domain-independent training data for dialogue context reconstruction, including time, location, people, and noun phrase examples.

Key Insights and Ideas

  • End-to-End is Better: Jointly training detection and resolution modules outperforms a step-by-step approach, mitigating the impact of cascading errors.
  • Data Augmentation is Crucial: The scarcity of annotated dialogue data is a major bottleneck. Their novel data generation method for CQA significantly boosts performance.
  • POS and Position Matter: Incorporating Part-of-Speech tags and positional embeddings is vital for understanding syntactic structure and resolving references effectively.
  • The UNK Token: Introducing an "UNK" (unknown) token for the null string in the resolution task helps the model identify cases where no suitable mention exists, improving robustness.
  • CNNs vs. LSTMs for Encoding: Convolutional layers (CNNs) proved more effective than LSTMs for the encoding layer, likely due to their ability to capture distant and global dependencies in coreference.

Key Results

The paper reports state-of-the-art performance on several benchmarks.

  • End-to-End Evaluation: Their end-to-end model achieved improved F1 scores compared to previous methods on the CQA dataset.
  • Zero Pronoun Resolution: On the CONLL2012 dataset, their POSNet-R model, after pretraining on CQA, achieved the best performance, outperforming existing methods.
  • Detection Performance: The POSNet-D component demonstrated high precision and recall for referring expression detection, indicating its ability to accurately identify candidates.

Limitations and Future Work

  • Data Size: While they generated a large CQA dataset, the paper acknowledges that more data is always beneficial.
  • Domain Specificity: The CQA dataset was generated to be domain-independent, but fine-tuning on domain-specific datasets is suggested for optimal performance.
  • Coreference Detection Bottleneck: The paper identifies zero pronoun detection as a bottleneck in the end-to-end system.
  • Future Directions: The authors suggest extending their work to wider contextual domains, incorporating more sophisticated linguistic studies of conjunctive relations, and integrating semantics from conversation history. They also propose exploring applications beyond context reconstruction, such as integrating their model into end-to-end question answering systems.

Summary updated

BibTeX
@inproceedings{yang2019end,
  author = {Wei Yang and Rui Qiao and Haocheng Qin and Amy Sun and Luchen Tan and Kun Xiong and Ming Li},
  title = {{End-to-End Neural Context Reconstruction in Chinese Dialogue}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure