Abstract
This paper describes our submission to the E2E NLG Challenge. Recently, neural seq2seq approaches have become mainstream in NLG, often resorting to pre- (respectively post-) processing delexicalization (relexicalization) steps at the word-level to handle rare words. By contrast, we train a simple character level seq2seq model, which requires no pre/post-processing (delexicalization, tokenization or even lowercasing), with surprisingly good results. For further improvement, we explore two re-ranking approaches for scoring candidates. We also introduce a synthetic dataset creation procedure, which opens up a new way of creating artificial datasets for Natural Language Generation.
Introduction
In Natural Language Generation (NLG), the primary objective is to transform structured Meaning Representations (MRs)—such as name[The Eagle], food[French]—into natural, human-like utterances. Traditional neural models typically operate at the word level. However, this approach necessitates "delexicalization," where specific entities are replaced with placeholders during training to manage out-of-vocabulary words. This process is often brittle, requiring extensive pre-processing and struggling with grammatical nuances like gender or number agreement. Furthermore, word-level models frequently suffer from omissions, failing to include all information provided in the source MR.
Methodology: The Char2Char Approach
We propose a "vanilla" character-to-character (char2char) sequence-to-sequence model. By processing input and output at the character level, the model learns the orthography of rare words directly, eliminating the need for tokenization, lowercasing, or delexicalization. This significantly simplifies the pipeline while maintaining high output quality.
Re-ranking Strategies
To further refine the output, we implemented two distinct re-ranking strategies to select the optimal sentence from the n-best list generated via beam search:
- The Reverse Model: A secondary model is trained to perform the inverse task: reconstructing the original MR from a generated utterance. Candidates are penalized based on the edit distance between the reconstructed MR and the actual input MR.
- The Classifier Re-ranker: A logistic regression classifier was developed to detect "semantic adequacy," identifying whether a model added or omitted information.
Synthetic Data Augmentation
To train the semantic adequacy classifier effectively, we introduced a novel synthetic data creation protocol. Rather than attempting to generate erroneous text, we manipulated existing correct pairs by "breaking" the MR—for example, by adding a slot to the MR that is absent in the corresponding text. This created a robust dataset of negative samples (omission errors) that allowed the classifier to learn the boundaries of semantic accuracy.
Experimental Results
Our system demonstrated strong performance in the E2E NLG Challenge. The "Forward" (vanilla) model achieved a BLEU score of 0.667, ranking 4th out of over 60 submissions.
The BLEU Paradox
An interesting finding was the inverse relationship between BLEU scores and semantic adequacy. While the re-ranking strategies improved the inclusion of all MR details (semantic accuracy), they often resulted in lower BLEU scores compared to the base model. This suggests that BLEU may favor sentences that "sound" like the average training example even if they omit critical information.
Human Evaluation
In human trials assessing "Quality" and "Naturalness," our system consistently placed in the second-best cluster among all competing systems, validating the effectiveness of the character-level approach.
Conclusion and Future Work
While the char2char model is highly effective, we observed occasional "copying" issues where the model defaulted to frequent restaurant names from the training set rather than the specific name in the MR. Future research will explore integrating the semantic classifier's score directly into the training process via Reinforcement Learning, potentially allowing the model to optimize for semantic adequacy during the generation phase itself.
Summary updated