A Large-Scale Corpus of E-mail Conversations with Standard and Two-Level Dialogue Act Annotations

Motoki Taniguchi, Yoshihiro Ueda, Tomoki Taniguchi, Tomoko OhkumaCOLING 2020

TL;DR

This paper introduces a large-scale, domain-agnostic corpus of e-mail conversations annotated with dialogue acts (DAs) using the ISO 24617-2 standard. They demonstrate that this corpus, along with multi-task learning on two-level DA annotations (sentence and message), significantly improves DA recognition models, even outperforming BERT on certain tasks and showing good generalization to other asynchronous conversation domains.

What Problem Does This Paper Address?

Understanding the communicative intent behind messages in online conversations is crucial for applications like summarization, question answering, and intelligent assistants. While dialogue act (DA) recognition aims to achieve this, existing corpora for asynchronous conversations (like emails and forums) suffer from limitations: they are often small-scale, use application-specific annotation schemes, and lack multi-level granularity (e.g., sentence vs. message level). This hinders the development of robust, generalizable DA recognition models. The paper addresses this by creating a large, standardized, and multi-level annotated corpus to enable better DA recognition for asynchronous communication.

Methods

The authors introduce a novel, large-scale corpus of e-mail conversations. They adopt the ISO 24617-2 standard for dialogue act annotation, ensuring domain and application independence. A key innovation is the two-level annotation scheme, where both sentences and messages are tagged with their primary communicative function. To evaluate their corpus and benchmark DA recognition, they implement and compare several models:

  • Handcrafted feature models: SVM, Logistic Regression (LR), and Conditional Random Fields (CRF).
  • Existing neural models: Bidirectional LSTM, Hierarchical LSTM (H-LSTM), and H-LSTM-CRF.
  • State-of-the-art model: BERT, a pre-trained contextual representation model, fine-tuned for DA recognition.

They also explore multi-task learning (MTL), where models are trained to predict both sentence-level and message-level DAs simultaneously, hypothesizing that this will improve learning efficiency.

Datasets

The primary dataset used is the Enron e-mail dataset, a publicly available collection of over six million messages. From this, they construct a corpus of over 2,000 threads, 6,000 messages, and 35,000 sentences. For comparison and domain adaptation experiments, they also utilize two existing corpora:

  • BC3: An e-mail corpus derived from mailing list threads.
  • QC3: A corpus of online forum conversations.

Key Insights and Ideas

  • Two-level annotation is powerful: Annotating both sentences and messages with DAs allows for more nuanced understanding and enables effective multi-task learning, leading to better model performance.
  • Large-scale, standardized data matters: Their large corpus, annotated with a domain-agnostic standard, proves crucial for achieving high performance and good generalization to other asynchronous conversation datasets.
  • BERT shows promise but isn't perfect: While BERT significantly outperforms previous neural models, it still falls short of human performance, indicating that DA recognition in this domain remains challenging.
  • Domain independence is achievable: Models trained on their corpus generalize well to other asynchronous conversation domains (BC3 and QC3), suggesting the ISO standard and large scale contribute to robustness.
  • Multi-task learning boosts performance: Training models to predict both sentence and message DAs simultaneously (MTL) significantly improves performance, especially with limited training data.

Key Results

  • BERT's Superiority: BERT achieved the best performance, outperforming previous state-of-the-art neural models by a significant margin (e.g., 8.5 points in sentence-level F1 score).
  • Human Performance Benchmark: Expert annotators achieved substantially higher F1 scores (e.g., 0.834 for sentence-level) than BERT, highlighting the difficulty of the task.
  • Domain Adaptation Success: Models trained on their corpus showed strong performance when transferred to BC3 and QC3, with ZERO-SHOT (training only on their corpus) even outperforming SELF (training only on the target domain) in some cases.
  • MTL Gains: Multi-task learning models (MTL-ENCODER and MTL-FULL) showed substantial gains over the SEPARATE model, particularly with limited data (e.g., 24.9 point increase in sentence-level F1 with 500 messages).
  • Imbalanced Data: The "Inform" tag was the most frequent, leading to an imbalanced distribution, which is a common challenge in DA recognition. Less frequent tags (fewer than 500 instances) were predicted less accurately.

Limitations and Future Work

  • Human Performance Gap: Despite BERT's strong performance, there remains a significant gap between automated models and human annotators, indicating room for further improvement.
  • Imbalanced Tag Distribution: The corpus exhibits an imbalanced distribution of dialogue act tags, which can negatively impact the performance on less frequent tags.
  • Future Work: The authors plan to explore end-to-end models that leverage the hierarchical structure of email conversations (words -> sentences -> messages -> threads). They also intend to extend annotations to include other DA components like rhetorical relations.

Summary updated

BibTeX
@inproceedings{taniguchi2020large,
  author = {Motoki Taniguchi and Yoshihiro Ueda and Tomoki Taniguchi and Tomoko Ohkuma},
  title = {{A Large-Scale Corpus of E-mail Conversations with Standard and Two-Level Dialogue Act Annotations}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
}
Figure