TL;DR
While end-to-end neural text generators are often assumed to handle both sentence planning and surface realization implicitly, this paper demonstrates that standard sequence-to-sequence models fail to perform fundamental discourse and sentence planning operations on their own. By introducing lightweight explicit supervision tokens (e.g., controlling sentence counts, distributive aggregation, or contrast), neural generators can dramatically improve semantic fidelity, gain stylistic control, and even generalize discourse operations to unseen attribute combinations.
What Problem Does This Paper Address?
Traditional natural language generation (NLG) pipelines separate generation into distinct stages: content planning, sentence planning (deciding sentence boundaries, combining clauses via aggregation, and structuring discourse relations like contrast or justification), and surface realization.
Recent end-to-end neural NLG (NNLG) systems collapsed these stages into a single model mapping meaning representations (MRs) straight to text. However, prior work largely evaluated fluency and slot error rate without testing whether neural generators actually learn to execute sentence planning operations. This paper investigates three critical questions:
- Can standard NNLG models perform common sentence planning operations?
- Can they decide whether to realize content across single or multiple sentences?
- Can they generalize discourse and aggregation operations to attribute values unseen together in training?
Methods
The authors build on TGen, an open-source sequence-to-sequence model with an LSTM encoder-decoder, additive attention, and an n-best reranker. They compare three supervision paradigms:
- No Supervision: The baseline input vector contains only standard MR dialogue acts and slot-value pairs.
- Token Supervision: The MR is supplemented with an explicit control token acting as a latent variable:
- Scoping: A
PERIODtoken specifying the exact target sentence count (or a reading complexity level: low, medium, high). - Aggregation: A binary
DISTRIBUTEtoken ($0$ or $1$) indicating whether two attributes sharing the same value (e.g., price and customer rating) should be merged into a single phrase (e.g., "high rating and cost"). - Contrast: A binary
CONTRASTtoken ($0$ or $1$) signaling whether to produce a discourse contrast (e.g., using "but", "although"). - Semantic Supervision (Distributive Aggregation only): The MR specifies the exact attribute value to distribute over (
LOW,AVERAGE,HIGH, orNONE).
Evaluation relies on automated regex/rule-based information extraction scripts measuring Slot Error Rate (SER) and operation-specific accuracy.
Datasets
Crowdsourced dialogue datasets like the E2E Generation Challenge (50k instances) contain very few instances of certain syntactic operations (e.g., exactly 1 instance of distributive aggregation).
To remedy this, the authors supplemented E2E data with synthetic, stylistically controlled generation from PERSONAGE using a New York City restaurant database. They released a systematically varied Sentence Planning NLG corpus of 204,955 utterance/MR pairs, configuring controlled subsets for:
- Sentence Scoping: 64,442 training pairs and 398 test pairs.
- Distributive Aggregation: 63,690 training pairs and 408 test pairs, intentionally holding out the combination
{PRICE=HIGH, RATING=HIGH}from training to evaluate generalization. - Discourse Contrast: Incremental training subsets (3K, 7K, 11K, and 21K instances) evaluated on E2E test sets and a 785-MR NYC test set.
Key Insights and Ideas
- Implicit planning fails: Without explicit control signals, standard sequence-to-sequence models struggle to infer sentence boundaries or aggregation logic from dialogue acts alone.
- Minimal tokens unlock mathematical generalization: A single binary token enabled the model to infer distributive equality and generalize it to the unseen
{PRICE=HIGH, RATING=HIGH}test condition with 98% accuracy. - More detail isn't always better: Counterintuitively, binary supervision far outperformed fine-grained semantic supervision for distributive aggregation (99% vs. 36% overall accuracy). Explicit string values confused the LSTM decoder, leading to repetitive stuttering (e.g., "high rating and high rating").
- Supervision curbs hallucination: Across all tasks, guiding the model's discourse structure also reduced Slot Error Rates, indicating that discourse confusion directly contributes to semantic errors.
Key Results
- Sentence Scoping: The unsupervised model achieved only 21.6% period accuracy (SER: 0.06). Adding the
PERIODcount token boosted accuracy to 99.5% (SER: 0.03; correlation with references: 0.998). In complexity classification (low/medium/high), it reached 89% accuracy. - Distributive Aggregation: Unsupervised models completely failed to generalize to the unseen
HIGHcombination (0.00% accuracy; 29% overall). Binary token supervision achieved 98% accuracy onHIGH(99% overall, SER: 0.07). Semantic supervision achieved only 9% onHIGH(36% overall, SER: 0.25). - Discourse Contrast: On the full 21K training set, the supervised
CONTRASTmodel increased contrast accuracy to 84% on E2E (vs. 50% unsupervised) and 75% on the E2E contrast-only test (vs. 62% unsupervised), while significantly increasing contrast attempts when semantically warranted (422 attempts vs. 13 without supervision).
Limitations and Future Work
- No Human Evaluation: The evaluation relies entirely on automated scripts for slot errors and discourse markers; output naturalness, fluency, and user preference were not evaluated via human trials.
- Edge-Case Sensitivity: In extreme scoping tests ($PERIOD = N - 1$), the model's accuracy dropped to 84%, showing vulnerability to rare training distributions.
- Domain Breadth: Experiments were confined to restaurant recommendations. Future work includes testing on larger, open-domain benchmarks (e.g., WebNLG, WikiBio) and exploring hierarchical representations for multi-attribute discourse plans.
Summary updated