Can Neural Generators for Dialogue Learn Sentence Planning and Discourse Structuring?

Lena Reed, Shereen Oraby, Marilyn A. Walker

TL;DR

While end-to-end neural text generators are often assumed to handle both sentence planning and surface realization implicitly, this paper demonstrates that standard sequence-to-sequence models fail to perform fundamental discourse and sentence planning operations on their own. By introducing lightweight explicit supervision tokens (e.g., controlling sentence counts, distributive aggregation, or contrast), neural generators can dramatically improve semantic fidelity, gain stylistic control, and even generalize discourse operations to unseen attribute combinations.

What Problem Does This Paper Address?

Traditional natural language generation (NLG) pipelines separate generation into distinct stages: content planning, sentence planning (deciding sentence boundaries, combining clauses via aggregation, and structuring discourse relations like contrast or justification), and surface realization.

Recent end-to-end neural NLG (NNLG) systems collapsed these stages into a single model mapping meaning representations (MRs) straight to text. However, prior work largely evaluated fluency and slot error rate without testing whether neural generators actually learn to execute sentence planning operations. This paper investigates three critical questions:

  1. Can standard NNLG models perform common sentence planning operations?
  2. Can they decide whether to realize content across single or multiple sentences?
  3. Can they generalize discourse and aggregation operations to attribute values unseen together in training?

Methods

The authors build on TGen, an open-source sequence-to-sequence model with an LSTM encoder-decoder, additive attention, and an n-best reranker. They compare three supervision paradigms:

  • No Supervision: The baseline input vector contains only standard MR dialogue acts and slot-value pairs.
  • Token Supervision: The MR is supplemented with an explicit control token acting as a latent variable:
  • Scoping: A PERIOD token specifying the exact target sentence count (or a reading complexity level: low, medium, high).
  • Aggregation: A binary DISTRIBUTE token ($0$ or $1$) indicating whether two attributes sharing the same value (e.g., price and customer rating) should be merged into a single phrase (e.g., "high rating and cost").
  • Contrast: A binary CONTRAST token ($0$ or $1$) signaling whether to produce a discourse contrast (e.g., using "but", "although").
  • Semantic Supervision (Distributive Aggregation only): The MR specifies the exact attribute value to distribute over (LOW, AVERAGE, HIGH, or NONE).

Evaluation relies on automated regex/rule-based information extraction scripts measuring Slot Error Rate (SER) and operation-specific accuracy.

Datasets

Crowdsourced dialogue datasets like the E2E Generation Challenge (50k instances) contain very few instances of certain syntactic operations (e.g., exactly 1 instance of distributive aggregation).

To remedy this, the authors supplemented E2E data with synthetic, stylistically controlled generation from PERSONAGE using a New York City restaurant database. They released a systematically varied Sentence Planning NLG corpus of 204,955 utterance/MR pairs, configuring controlled subsets for:

  • Sentence Scoping: 64,442 training pairs and 398 test pairs.
  • Distributive Aggregation: 63,690 training pairs and 408 test pairs, intentionally holding out the combination {PRICE=HIGH, RATING=HIGH} from training to evaluate generalization.
  • Discourse Contrast: Incremental training subsets (3K, 7K, 11K, and 21K instances) evaluated on E2E test sets and a 785-MR NYC test set.

Key Insights and Ideas

  • Implicit planning fails: Without explicit control signals, standard sequence-to-sequence models struggle to infer sentence boundaries or aggregation logic from dialogue acts alone.
  • Minimal tokens unlock mathematical generalization: A single binary token enabled the model to infer distributive equality and generalize it to the unseen {PRICE=HIGH, RATING=HIGH} test condition with 98% accuracy.
  • More detail isn't always better: Counterintuitively, binary supervision far outperformed fine-grained semantic supervision for distributive aggregation (99% vs. 36% overall accuracy). Explicit string values confused the LSTM decoder, leading to repetitive stuttering (e.g., "high rating and high rating").
  • Supervision curbs hallucination: Across all tasks, guiding the model's discourse structure also reduced Slot Error Rates, indicating that discourse confusion directly contributes to semantic errors.

Key Results

  • Sentence Scoping: The unsupervised model achieved only 21.6% period accuracy (SER: 0.06). Adding the PERIOD count token boosted accuracy to 99.5% (SER: 0.03; correlation with references: 0.998). In complexity classification (low/medium/high), it reached 89% accuracy.
  • Distributive Aggregation: Unsupervised models completely failed to generalize to the unseen HIGH combination (0.00% accuracy; 29% overall). Binary token supervision achieved 98% accuracy on HIGH (99% overall, SER: 0.07). Semantic supervision achieved only 9% on HIGH (36% overall, SER: 0.25).
  • Discourse Contrast: On the full 21K training set, the supervised CONTRAST model increased contrast accuracy to 84% on E2E (vs. 50% unsupervised) and 75% on the E2E contrast-only test (vs. 62% unsupervised), while significantly increasing contrast attempts when semantically warranted (422 attempts vs. 13 without supervision).

Limitations and Future Work

  • No Human Evaluation: The evaluation relies entirely on automated scripts for slot errors and discourse markers; output naturalness, fluency, and user preference were not evaluated via human trials.
  • Edge-Case Sensitivity: In extreme scoping tests ($PERIOD = N - 1$), the model's accuracy dropped to 84%, showing vulnerability to rare training distributions.
  • Domain Breadth: Experiments were confined to restaurant recommendations. Future work includes testing on larger, open-domain benchmarks (e.g., WebNLG, WikiBio) and exploring hierarchical representations for multi-attribute discourse plans.

Summary updated

Abstract

Responses in task-oriented dialogue systems often realize multiple propositions whose ultimate form depends on the use of sentence planning and discourse structuring operations. For example a recommendation may consist of an explicitly evaluative utterance e.g. Chanpen Thai is the best option, along with content related by the justification discourse relation, e.g. It has great food and service, that combines multiple propositions into a single phrase. While neural generation methods integrate sentence planning and surface realization in one end-to-end learning framework, previous work has not shown that neural generators can: (1) perform common sentence planning and discourse structuring operations; (2) make decisions as to whether to realize content in a single sentence or over multiple sentences; (3) generalize sentence planning and discourse relation operations beyond what was seen in training. We systematically create large training corpora that exhibit particular sentence planning operations and then test neural models to see what they learn. We compare models without explicit latent variables for sentence planning with ones that provide explicit supervision during training. We show that only the models with additional supervision can reproduce sentence planning and discourse operations and generalize to situations unseen in training.
BibTeX
@inproceedings{reed2018can,
  author = {Lena Reed and Shereen Oraby and Marilyn A. Walker},
  title = {{Can Neural Generators for Dialogue Learn Sentence Planning and Discourse Structuring?}},
  booktitle = {INLG},
  year = {2018},
  url = {https://openreview.net/forum?id=YzPwhK00TKt},
}

Loading related content...

Figure