Predicting Discourse Structure using Distant Supervision from Sentiment

Patrick Huber, Giuseppe Carenini

TL;DR

This paper tackles the scarcity of annotated data for discourse structure parsing by proposing a novel "distant supervision" approach. They leverage readily available document-level sentiment labels to automatically generate a large-scale dataset of discourse trees, which then enables training more robust discourse parsers, especially for cross-domain applications.

What Problem Does This Paper Address?

Discourse parsing, the task of understanding the hierarchical structure of text and the relationships between its parts, is crucial for many downstream NLP tasks like sentiment analysis and summarization. However, progress in this field has been hampered by a lack of large, annotated datasets. Existing corpora are small and domain-specific, limiting the effectiveness of modern deep learning models that thrive on abundant data. Creating these annotations is also expensive and time-consuming, requiring linguistic expertise. This paper aims to bridge this gap by creating a massive dataset of discourse structures without manual annotation.

Methods

The core of the proposed method is a two-stage process:

  1. Distant Supervision with Multiple-Instance Learning (MIL): The authors adapt a Multiple-Instance Learning (MIL) approach, specifically MILNet, to learn fine-grained sentiment and "attention" scores for individual text segments (Elementary Discourse Units or EDUs). Instead of direct EDU-level labels, they use document-level sentiment labels (e.g., positive/negative review) as the "bag" label. The MILNet, trained on this document-level supervision, learns to infer the contribution of each EDU to the overall document sentiment and its importance. This stage leverages pre-trained word embeddings and recurrent neural networks with attention mechanisms.
  2. CKY-style Tree Generation: The inferred EDU-level sentiment and attention scores are then fed into a CKY-style dynamic programming algorithm. This algorithm constructs discourse trees by aggregating information from child nodes to parent nodes. The key is that the tree generation is optimized to best predict the original document-level sentiment. Different aggregation functions for sentiment and attention scores are explored (average, maximum, sum) to find the best way to combine these signals. The output is a complete RST-style discourse tree for each document.

Datasets

  • Yelp'13: A large dataset of restaurant reviews with star ratings, used for training the MILNet model. The authors generated 100,000 discourse trees from this dataset.
  • RST-DT: The Rhetorical Structure Theory Discourse Treebank, a standard corpus of news articles. Used for evaluating the trained discourse parsers.
  • Instr-DT: The Instructional Dataset, a corpus of home-repair instructions. Also used for evaluation.

The paper introduces a new, automatically generated dataset of 100,000 discourse trees derived from the Yelp'13 corpus, which is two orders of magnitude larger than existing corpora.

Key Insights and Ideas

  • Bidirectional Synergy: The paper validates the intuition that discourse structure and sentiment analysis can mutually benefit each other. Here, they demonstrate that sentiment analysis can be a powerful auxiliary task to generate discourse structure data.
  • MIL for Discourse: Adapting MIL for discourse parsing is a clever way to leverage readily available document-level labels when fine-grained, EDU-level annotations are scarce.
  • CKY for Optimization: Using a CKY-style algorithm to build trees that optimize for document-level sentiment provides a principled way to select the "best" discourse structure from many possibilities.
  • Inter-Domain Transfer: The generated dataset shows significant promise for improving discourse parsers' performance when transferred to new, unseen domains, which is a major practical challenge.

Key Results

  • A discourse parser trained on the automatically generated Yelp'13 discourse tree dataset achieves strong performance on inter-domain discourse structure prediction, outperforming parsers trained on smaller, human-annotated datasets (RST-DT and Instr-DT) when tested on those domains.
  • While the parser trained on the new dataset doesn't quite match the performance of a parser trained and tested on the same domain (intra-domain), its inter-domain performance is remarkably close to intra-domain baselines.
  • The "average" attention aggregation function generally yields the most consistent performance across different domains.
  • Performance generally increases with the size of the training dataset, highlighting the value of the large, automatically generated corpus.

Limitations and Future Work

  • Nuclearity and Relation Labels: The generated discourse trees primarily focus on structure. The paper acknowledges that assigning nuclearity and rhetorical relation labels is left for future work.
  • Neural Parsers: The current experiments use a state-of-the-art SVM-based parser. The authors plan to experiment with neural discourse parsers on their generated dataset, as these have historically underperformed due to data limitations.
  • End-to-End System: A future direction is to develop an end-to-end system that jointly learns discourse structure generation and parsing, rather than a two-stage approach.
  • Domain Specificity: While the new dataset improves inter-domain transfer, performance still varies by domain, with the Instr-DT corpus showing lower performance, possibly due to shorter documents.

Summary updated

Abstract

Discourse parsing could not yet take full advantage of the neural NLP revolution, mostly due to the lack of annotated datasets. We propose a novel approach that uses distant supervision on an auxiliary task (sentiment classification), to generate abundant data for RST-style discourse structure prediction. Our approach combines a neural variant of multiple-instance learning, using document-level supervision, with an optimal CKY-style tree generation algorithm. In a series of experiments, we train a discourse parser (for only structure prediction) on our automatically generated dataset and compare it with parsers trained on human-annotated corpora (news domain RST-DT and Instructional domain). Results indicate that while our parser does not yet match the performance of a parser trained and tested on the same dataset (intra-domain), it does perform remarkably well on the much more difficult and arguably more useful task of inter-domain discourse structure prediction, where the parser is trained on one domain and tested/applied on another one.
BibTeX
@inproceedings{huber2019predicting,
  author = {Patrick Huber and Giuseppe Carenini},
  title = {{Predicting Discourse Structure using Distant Supervision from Sentiment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1235},
}
Figure