Dialog Acts Classification for Question-Answer Corpora

Saurabh Chakravarty, Raja Venkata Satya Phanindra Chava, Edward A. FoxASAIL@ICAIL 2019 · OpenReview

TL;DR

This paper tackles the challenge of applying NLP to question-answer (QA) corpora, like legal depositions, by developing a dialog act classification system. They introduce a new dialog act ontology and leverage deep learning models (CNN, LSTM, BERT) to classify utterances, achieving a significant F1 score of 0.84 with BERT, paving the way for better downstream NLP tasks.

What Problem Does This Paper Address?

Legal depositions, while rich in information, present a significant challenge for standard Natural Language Processing (NLP) techniques. These documents are structured as rapid-fire question-answer (QA) pairs, often featuring incomplete sentences, informal language, and frequent context shifts. This makes tasks like summarization or information extraction difficult because the inherent structure and nuances of conversational language are not well-handled by traditional NLP methods. The authors aim to bridge this gap by classifying the communicative intent (dialog act) of each utterance, thereby enabling more sophisticated NLP applications on such corpora.

Methods

The core of the paper lies in developing a robust dialog act classification system. The authors first define a Dialog Act ontology specifically tailored for legal depositions, expanding upon existing frameworks to capture the nuances of questions and answers in this domain. This ontology includes 20 distinct dialog acts.

For classification, they explore three deep learning architectures:

  • Convolutional Neural Network (CNN): This approach uses CNNs to capture n-gram representations of sentences, leveraging word2vec embeddings.
  • Bi-directional Long Short-Term Memory (LSTM) with Attention: This method utilizes LSTMs to capture long-term dependencies in sentences, enhanced by an attention mechanism to focus on important parts of the utterance.
  • BERT (Bidirectional Encoder Representations from Transformers): This state-of-the-art model is fine-tuned for single-sentence classification. The authors use pre-trained BERT embeddings and add a feed-forward neural network layer for the classification task.

The models were trained and evaluated using a manually annotated dataset derived from legal depositions.

Datasets

The primary dataset used in this research is a proprietary dataset consisting of approximately 350 legal depositions, provided by Mayfair Group LLC. This dataset was manually annotated for dialog act classification.

Additionally, the authors mention the Tobacco Dataset, a publicly available corpus of around 12,300 legal deposition documents from the UCSF Library, which could be used for reproduction of their analysis.

Key Insights and Ideas

  • Dialog Acts are Crucial for QA Corpora: The paper strongly argues that understanding the communicative intent (dialog act) of each utterance is a fundamental step for effectively processing QA-heavy documents like legal depositions.
  • Domain-Specific Ontology is Key: Developing a dialog act ontology tailored to the specific characteristics of legal depositions, rather than relying solely on general conversation ontologies, is essential for accurate classification.
  • BERT's Superiority: The research highlights BERT's significant advantage over traditional CNN and LSTM models for this specific task, demonstrating its power in capturing complex linguistic nuances.
  • Potential for Downstream Tasks: By accurately classifying dialog acts, the paper lays the groundwork for applying advanced NLP techniques like summarization and question answering to these challenging corpora.

Key Results

The paper reports the following F1 scores for the tested classifiers on the dialog act classification task:

  • BERT: 0.84
  • LSTM: 0.71
  • CNN: 0.57

BERT significantly outperformed both CNN and LSTM, achieving the highest F1 score. The authors also observed that after about 15 epochs, the validation accuracy for all models plateaued, indicating that further training might not yield significant improvements.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Limited Training Data for Certain Classes: Some dialog acts, like "confront," suffer from a lack of training data, leading to misclassifications. The authors suggest adding more specific training data for these classes.
  • Subtlety Between Classes: There's a subtle difference between some dialog acts (e.g., "bin" and "bin-d") that the classifier sometimes struggles to distinguish.
  • Annotation Errors: Some misclassifications were identified as potential annotation errors, suggesting the need for careful review of the ground truth.
  • Merging Similar Classes: The "so" and "sno" classes are very similar and could potentially be merged in future work.

Future work includes:

  • Contextual Dialog Act Classification: Incorporating dialog acts from previous utterances to improve classification accuracy.
  • Developing Representation Techniques: Creating semantically equivalent representations for QA pairs that are easier for NLP tools to process.
  • Abstractive Summarization: Using these representations to generate summaries via advanced deep learning methods.
  • Explainable AI: Developing methods to explain how summaries are generated.

Summary updated

BibTeX
@inproceedings{chakravarty2019dialog,
  author = {Saurabh Chakravarty and Raja Venkata Satya Phanindra Chava and Edward A. Fox},
  title = {{Dialog Acts Classification for Question-Answer Corpora}},
  booktitle = {ASAIL@ICAIL},
  year = {2019},
  url = {https://openreview.net/forum?id=GmuKDvqS3K},
}
Figure