Can Neural Generators for Dialogue Learn Sentence Planning and Discourse Structuring?

Lena Reed, Shereen Oraby, Marilyn A. Walker

Abstract

Responses in task-oriented dialogue systems often realize multiple propositions whose ultimate form depends on the use of sentence planning and discourse structuring operations. For example a recommendation may consist of an explicitly evaluative utterance e.g. Chanpen Thai is the best option, along with content related by the justification discourse relation, e.g. It has great food and service, that combines multiple propositions into a single phrase. While neural generation methods integrate sentence planning and surface realization in one end-to-end learning framework, previous work has not shown that neural generators can: (1) perform common sentence planning and discourse structuring operations; (2) make decisions as to whether to realize content in a single sentence or over multiple sentences; (3) generalize sentence planning and discourse relation operations beyond what was seen in training. We systematically create large training corpora that exhibit particular sentence planning operations and then test neural models to see what they learn. We compare models without explicit latent variables for sentence planning with ones that provide explicit supervision during training. We show that only the models with additional supervision can reproduce sentence planing and discourse operations and generalize to situations unseen in training.

TL;DR

While neural dialogue generators are often marketed as "end-to-end" solutions, they struggle to learn complex sentence planning and discourse structuring on their own. This paper demonstrates that by adding simple "supervision tokens"—explicit markers for sentence counts or discourse relations—models can achieve near-perfect structural control and, crucially, generalize to linguistic patterns they never saw during training.

What Problem Does This Paper Address?

In task-oriented dialogue (like a restaurant recommender), a system must convert a structured Meaning Representation (MR)—a list of attributes like NAME[Zizzi], FOOD[English], PRICE[moderate]—into a natural-sounding response.

Traditionally, this was a pipeline: first, a sentence planner decided how to group information (scoping) and link ideas (discourse relations), then a surface realizer turned those plans into words. Modern Neural Natural Language Generation (NNLG) models try to collapse this into a single step. However, it wasn't clear if these "black box" models actually learned the underlying logic of planning.

Examples of sentence scoping: the same meaning representation (MR) can be realized as 1, 3, or 5 sentences.

Do they know how to contrast a positive attribute with a negative one? Can they decide to split a long list into exactly three sentences? The authors argue that without explicit guidance, neural models often fail to produce semantically accurate or structurally varied outputs.

Methods

The researchers used a standard sequence-to-sequence (seq2seq) architecture with LSTMs and attention (specifically the TGen framework). The core of their experiment lies in comparing three levels of supervision:

  1. No Supervision: The model receives only the MR attributes.
  2. Token Supervision: The input is supplemented with a "supervision token" (e.g., a number representing the desired sentence count, or a binary flag for "contrast").
  3. Semantic Supervision: (Used for aggregation) The token provides specific instructions on which values to group together.
The neural architecture showing how supervision tokens are integrated into the input vector to guide the generator.

To ensure the models had enough "clean" examples to learn from, the authors used PERSONAGE, a rule-based generator, to create a massive synthetic dataset where sentence planning operations were perfectly labeled.

Datasets

  • E2E Generation Challenge: A standard crowdsourced dataset of 50k restaurant descriptions. The authors found this dataset lacked enough stylistic variety (e.g., only one instance of "distributive aggregation").
  • PERSONAGE Synthetic Data: 204,955 utterance/MR pairs systematically varied to exhibit specific planning operations like scoping and aggregation.
  • NYC Restaurant Dataset: Used specifically for discourse contrast experiments, providing a more diverse set of attributes to compare and contrast.

Key Insights and Ideas

  • The "End-to-End" Myth: Simply providing more data isn't enough for a model to induce the "mathematical" logic of language, such as the distributive property (e.g., realizing PRICE[low] and RATING[low] as "low price and rating").
  • Supervision Enables Generalization: This is the "aha moment." Models with supervision tokens didn't just memorize training examples; they learned the operation. For instance, a model trained to group "low" values could successfully group "high" values it had never seen in a distributive context before.
  • Control Improves Accuracy: Adding structural supervision actually reduced "hallucinations" (making things up) and "omissions" (forgetting attributes). By telling the model how to structure the sentence, it became better at remembering what to say.

Key Results

  • Sentence Scoping: The "No Supervision" model correctly predicted the number of sentences only 22% of the time. With a "Period Count" token, accuracy jumped to 99.5%.
  • Distributive Aggregation: When asked to group attributes with a value ("HIGH") never seen in training, the unsupervised model failed completely (0% accuracy). The model with binary supervision handled it with 98% accuracy.
  • Discourse Contrast: On the E2E test set, adding a CONTRAST token improved the correct usage of contrastive markers (like "but" or "however") from 50% to 84%, while simultaneously lowering the Slot Error Rate (SER).

Limitations and Future Work

The authors acknowledge that their evaluation focused heavily on semantic accuracy and structural control rather than human-perceived "naturalness" or "fluency." While the models became highly controllable, the study doesn't measure if a human would find the resulting 5-sentence response more annoying than a 1-sentence one.

Future directions include testing these methods on more complex real-world datasets and investigating hierarchical input representations to better mirror the complex planning humans do when speaking.

Summary updated

Abstract

Responses in task-oriented dialogue systems often realize multiple propositions whose ultimate form depends on the use of sentence planning and discourse structuring operations. For example a recommendation may consist of an explicitly evaluative utterance e.g. Chanpen Thai is the best option, along with content related by the justification discourse relation, e.g. It has great food and service, that combines multiple propositions into a single phrase. While neural generation methods integrate sentence planning and surface realization in one end-to-end learning framework, previous work has not shown that neural generators can: (1) perform common sentence planning and discourse structuring operations; (2) make decisions as to whether to realize content in a single sentence or over multiple sentences; (3) generalize sentence planning and discourse relation operations beyond what was seen in training. We systematically create large training corpora that exhibit particular sentence planning operations and then test neural models to see what they learn. We compare models without explicit latent variables for sentence planning with ones that provide explicit supervision during training. We show that only the models with additional supervision can reproduce sentence planing and discourse operations and generalize to situations unseen in training.
BibTeX
@inproceedings{reed2018can,
  author = {Lena Reed and Shereen Oraby and Marilyn A. Walker},
  title = {{Can Neural Generators for Dialogue Learn Sentence Planning and Discourse Structuring?}},
  booktitle = {INLG},
  year = {2018},
  url = {https://openreview.net/forum?id=YzPwhK00TKt},
}
Figure