TL;DR
This paper tackles the problem of errors in neural natural language generation (NLG) by proposing a novel symbolic intermediate representation. By breaking down the generation process into distinct planning and surface realization stages, they achieve state-of-the-art results on the E2E dataset and demonstrate the potential of pipeline approaches in neural NLG.
What Problem Does This Paper Address?
Neural Natural Language Generation (NLG) systems are adept at producing fluent and human-like text. However, they often suffer from critical errors such as hallucination (generating information not present in the input), repetition, and contradictions, leading to a lack of adequacy. Traditional rule-based NLG systems, while more reliable in terms of content, are labor-intensive to create and maintain. This paper aims to bridge this gap by developing a method that leverages the fluency of neural models while mitigating their error-prone nature, particularly in ensuring the generated text is adequate and accurate. They propose a multi-stage pipeline approach that separates the "what to say" (planning) from the "how to say it" (surface realization), using a structured intermediate representation to facilitate this separation.
Methods
The core of this work lies in designing a symbolic intermediate representation (IR). Instead of a fully end-to-end neural model, they adopt a two-stage pipeline:
- Utterance Planning: A neural model takes structured input data and generates an IR.
- Surface Realization: Another neural model takes this IR and generates the final natural language text.
The chosen IR is based on the Underspecified Universal Dependency (UUD) structure. This structure represents a sentence as a tree of content words linked by predicate-argument edges, inspired by linguistic dependency parsing. To make it suitable for neural sequence-to-sequence models, they:
- Linearize the UUD tree using a depth-first search order, adding scope markers.
- Remove auxiliary token features (like part-of-speech tags and detailed lexical features) that are typically present in UUD structures.
- Use the original form of tokens rather than lemmas, simplifying the task for the surface realization model.
This approach is driven by three key constraints: the IR must be processable by neural systems, the surface realization task should not be overly difficult, and it should be possible to parse existing sentences into this IR to augment training data.
Datasets
- E2E Dataset: Used for both training and evaluation of the end-to-end system. This dataset consists of meaning representations and corresponding utterances.
- TripAdvisor Corpus: Used to augment the training data for the surface realization model. This corpus was filtered to match the E2E vocabulary and sentence length constraints.
Key Insights and Ideas
- Pipeline Power: Splitting NLG into distinct planning and surface realization stages, even with neural models, can lead to improved performance and potentially better control over output quality.
- Symbolic IR as a Bridge: A well-designed symbolic IR can effectively decouple semantic planning from syntactic realization, allowing for more modular and potentially more robust generation systems.
- Data Augmentation Benefits: Augmenting the surface realization model's training data with a larger, related corpus (TripAdvisor) significantly improves its performance, highlighting the value of data diversity.
- Simplification for Neural Models: By simplifying the UUD structure (e.g., using original token forms instead of lemmas and removing detailed features), the surface realization task becomes more manageable for neural models.
Key Results
- Surface Realization Quality: The surface realization model, when trained on the E2E dataset, achieved a BLEU score of 82.47. Augmenting with the TripAdvisor corpus increased this to 83.38.
- Manual Evaluation: Manual analysis showed that in only a small proportion (~5%) of outputs, the meaning of the reference sentence was not fully recovered.
- End-to-End Performance: The full pipeline system (utterance planning + surface realization) outperformed the winning system of the E2E challenge on automated metrics.
- Comparison to Baselines: The proposed pipeline system (both with and without TripAdvisor data augmentation) achieved higher BLEU, NIST, METEOR, ROUGE, and CIDEr scores compared to baseline systems like TGen and Slug2Slug on the E2E test set.
Limitations and Future Work
- Manual Analysis Depth: While manual analysis was performed, the authors suggest further human evaluation is needed to fully understand the meaningfulness of the higher automated results and to pinpoint specific failure modes.
- Simplifications in IR: The IR omits certain features (like token features) and uses original token forms instead of lemmas. While this simplifies the task, it might limit the surface realization model's ability to reconstruct complex linguistic phenomena.
- Semantic Reranker: The authors note that other high-performing systems often use a semantic reranker. Adding such a component to their utterance planning step could potentially improve performance.
- Abstract IR for Rule-Based Systems: The current IR is designed for neural models. They suggest that a more abstract IR might be preferable for rule-based utterance planning systems, opening avenues for hybrid approaches.
- Token Features Generation: Generating token features (like POS tags) using multitask learning is mentioned as a potential future direction.
Summary updated