TL;DR
This paper presents the ADAPT system for the Surface Realization Shared Task 2020, demonstrating how effective data preprocessing and augmentation can significantly boost neural model performance. By carefully transforming input linguistic structures and expanding the training data with automatically parsed corpora, they achieved state-of-the-art results, highlighting the crucial role of these techniques in natural language generation.
What Problem Does This Paper Address?
Surface realization is the final, critical step in Natural Language Generation (NLG) systems. Its goal is to convert structured data or semantic representations into grammatically correct and natural-sounding sentences. While neural models have shown promise, achieving high accuracy in this task, especially for real-world applications like dialogue systems or personalized marketing, remains a challenge. This paper tackles how to improve the performance of neural surface realization systems, specifically by investigating the impact of preprocessing techniques and data augmentation strategies.
Methods
The ADAPT system employs a neural sequence-to-sequence model built upon a fork of the OpenNMT-py framework. Key methodological contributions include:
- Depth-First Linearization with Randomization: Input linguistic structures (parse trees) are linearized using a depth-first traversal. To ensure robustness, multiple random linearizations are generated for each sentence, introducing variability in the input sequence.
- Token-Level Features: Each token in the linearized sequence is augmented with several features: lemma, XPOS (eXtended Part-of-Speech tag), token ID, head of the token, dependency relation (DepRel), and linearization index (Lin). This allows for factored sequence models.
- Scoping Brackets: Brackets are added around child nodes in the linearized tree to provide explicit structural information to the model, even within a linear sequence.
- Form Suggestions: To address the challenge of generating correct word forms from lemmas, the system uses a dictionary of "form suggestions." This dictionary maps lemma-XPOS pairs to lists of observed word forms from a parsed corpus, helping the model predict the correct inflection.
- Augmented Dataset: A significant portion of the training data is generated by parsing sentences from large, publicly available corpora (Wikitext 103 and CNN stories). This augmentation aims to expose the model to a wider range of linguistic phenomena.
- Restricted Beam Search: During decoding, the beam search is constrained to consider only tokens present in the input sequence or those already generated in the output, aiming to reduce errors.
- Copy Attention: This mechanism allows the model to directly copy tokens from the source sequence to the generated text, which is crucial for handling out-of-vocabulary words.
Datasets
- Universal Dependencies English Web Treebank (EWT): This is the primary dataset used for training and evaluation.
- Augmented Datasets: Sentences parsed from:
- Wikitext 103: A large language modeling dataset.
- CNN stories: A dataset of news articles and their summaries.
The augmented datasets are created by parsing sentences from these corpora and then applying the same preprocessing steps as the EWT data.
Key Insights and Ideas
- Preprocessing is Paramount: The paper strongly suggests that sophisticated preprocessing, particularly linearization strategies and the inclusion of rich token-level features, is as important, if not more so, than the neural architecture itself for surface realization.
- Data Augmentation's Power: Augmenting the training data with automatically parsed sentences from large corpora significantly improves performance across various metrics, demonstrating the value of diverse training examples.
- Robustness through Randomization: Generating multiple random linearizations of parse trees helps the model learn to be robust to variations in input structure, leading to better generalization.
- Form Generation is a Key Challenge: The explicit inclusion of "form suggestions" highlights that predicting correct word inflections is a non-trivial aspect of surface realization that requires dedicated handling.
Key Results
The paper reports strong performance on the SR'20 test set:
- Automated Metrics (EWT Test Set): The system trained with the augmented corpora achieved a BLEU score of 87.5, NIST of 13.81, and DIST of 90.35, outperforming the system trained solely on EWT (BLEU 80.4, NIST 13.47, DIST 85.5).
- Human Evaluation (Readability): The augmented system achieved a readability score of 92.6, nearly matching the original human text's readability (90.7), indicating that the generated sentences are fluent and well-formed.
- Human Evaluation (Meaning Similarity): The augmented system achieved an average meaning similarity of 92.6%, a significant improvement and a strong indicator of the system's ability to preserve the original meaning.
Limitations and Future Work
- Meaning Similarity Goal: While 92.6% meaning similarity is high, the ultimate goal is 100% to ensure reliability for real-world NLG systems. The authors acknowledge that achieving perfect meaning preservation remains an ongoing challenge.
- Automatic Data Issues: The paper notes that automatically created data can be problematic. However, their approach of parsing existing sentences, rather than learning from rule-based systems, is presented as a way to mitigate this by learning to reverse the parsing process.
- Future Work: The authors implicitly suggest that further exploration of preprocessing techniques and more sophisticated data augmentation strategies could lead to even greater improvements. The focus on achieving higher meaning similarity also points towards future research directions.
Summary updated