A Joint Model of Rhetorical Discourse Structure and Summarization

Naman Goyal, Jacob EisensteinSPNLP @ EMNLP 2016 · OpenReview

TL;DR

This paper introduces a novel approach to discourse parsing, which aims to understand the hierarchical structure of text, by jointly modeling it with text summarization. By leveraging the SampleRank algorithm, they explore a more global inference strategy and investigate training discourse parsers using only document-summary pairs, a form of indirect supervision. While the direct supervised approach showed mixed results, the indirect supervision method, aiming to train a discourse parser by optimizing summary quality, did not outperform a simple baseline.

What Problem Does This Paper Address?

Understanding the structure of text beyond individual sentences is crucial for many natural language processing tasks, such as summarization, question answering, and dialogue systems. Rhetorical Structure Theory (RST) provides a framework for this, modeling text as a tree where segments are related by nuclearity (nucleus and satellite). However, existing RST discourse parsers face two main challenges:

  1. Local Inference Limitations: Many parsers rely on local inference techniques that don't capture document-level structural regularities, potentially missing important global patterns.
  2. Data Scarcity: Manually annotating discourse structures is expensive and time-consuming, leading to limited training data for discourse parsers.

This paper also explores the connection between discourse structure and summarization, noting that the nucleus of a discourse relation often indicates importance for a summary. This suggests that a good discourse parser could aid summarization, and conversely, that summarization quality might serve as a proxy for good discourse structure.

Methods

The core of this paper's methodological contribution lies in the application and adaptation of the SampleRank structure learning algorithm.

  • SampleRank for Discourse Parsing: SampleRank is a stochastic structure learning algorithm that allows for:
  • Arbitrary Features: It can incorporate document-level features that go beyond local context, addressing the first problem.
  • Global Inference: It uses a Metropolis-Hastings algorithm to explore the space of possible discourse structures, enabling a more global view.
  • Learning Objective: It can be trained to directly optimize a desired objective (like F1 score for span and nuclearity) rather than a proxy like log-likelihood.
  • Joint Model of Discourse Structure and Summarization: To tackle the data scarcity problem, the authors propose a joint model. They introduce summaries as an additional variable and use a constraint that links the summary to the discourse structure: elements aligned with the summary must be close to the root of the RST tree. This allows for indirect supervision, where the model is trained to produce good summaries, implicitly learning a compatible discourse structure.
  • Dependency Discourse Trees (DEP-DT): To facilitate sampling and constraint satisfaction, RST nuclearity structures are converted into DEP-DTs, which are spanning trees over elementary discourse units (EDUs).
  • Proposal Distribution: The sampling process involves proposing new discourse structures and summaries. The discourse structure sampler is based on local changes (edge polarity changes, tree rotations) to ensure ergodicity. The summary sampler generates summaries that adhere to the DEP-DT constraints.

Datasets

  • RST Discourse Treebank (RST-DT): Used for supervised evaluation of the discourse parsing model. It contains 385 documents, split into 347 for training and 38 for testing.
  • CNN News Articles and Summaries: A larger dataset of 2000 news articles and their bullet-point summaries scraped from the CNN website. This dataset was used for training the joint model under indirect supervision. The summaries are noted as not necessarily extractive.

Key Insights and Ideas

  • Global Inference for Discourse Parsing: The paper highlights the potential of global inference methods like SampleRank to capture document-level structural regularities that local methods might miss.
  • Indirect Supervision for Discourse Parsing: The core idea of using summarization quality as a proxy for good discourse structure is a novel way to overcome the data scarcity issue in discourse parsing. This leverages readily available summary data.
  • The Interplay of Discourse and Summarization: The work reinforces the known connection between discourse structure (especially nuclearity) and text importance for summarization, proposing a joint learning framework to exploit this.
  • SampleRank's Flexibility: The paper demonstrates SampleRank's ability to handle arbitrary features and optimize complex objectives, making it a promising tool for structured prediction tasks.

Key Results

  • Supervised Evaluation:
    • The SampleRank-based parser achieved competitive results on the span prediction task, outperforming most baselines except for a CRF approach that used rich linguistic features.
  • On the nuclearity prediction task, the SampleRank parser performed somewhat worse than prior efforts.
  • Indirect Supervision Evaluation:
    • The joint model trained using indirect supervision (optimizing summary quality) failed to outperform a simple "LEAD" baseline (selecting the first n sentences) on ROUGE scores for summarization.
  • Even when using the Tree Knapsack Problem (TKP) summarization algorithm with supervised SampleRank parses, the performance did not surpass the LEAD baseline.

Limitations and Future Work

  • Mixed Results in Supervised Setting: The authors acknowledge mixed results in the fully supervised case, particularly on the nuclearity metric. They suggest that fine-tuning the training objective might help balance these metrics.
  • Failure of Indirect Supervision Baseline: The most significant limitation is the failure of the indirectly supervised model to outperform a simple baseline in summarization. This suggests that either the learned discourse structures were not sufficiently informative for summarization, or the summarization features and constraints were not expressive enough.
  • Future Work:
    • More Expressive Features: The authors suggest designing more sophisticated features for capturing summarization quality to improve the joint model.
  • Joint Model with Labeled Parses and Summaries: Learning a joint model from both labeled discourse parses and summaries could be a more direct path to improvement.
  • Exploring Other Indirect Supervision Tasks: The concept of indirect supervision could potentially be applied to other downstream tasks beyond summarization.
  • Stochastic Search Limitations: The paper notes that while stochastic search offers advantages, it is not guaranteed to fully explore the search space in finite time.

Summary updated

BibTeX
@inproceedings{goyal2016joint,
  author = {Naman Goyal and Jacob Eisenstein},
  title = {{A Joint Model of Rhetorical Discourse Structure and Summarization}},
  booktitle = {SPNLP @ EMNLP},
  year = {2016},
  url = {https://openreview.net/forum?id=B1VmQfzdbB},
}
Figure