Abstract
This paper evaluates the performance of neural encoder-decoder models against rule-based template systems in the context of the E2E Natural Language Generation (NLG) Challenge. While neural models achieve higher scores on automatic metrics like BLEU, they often inherit noise from crowd-sourced training data, leading to factual omissions and grammatical errors. In contrast, a simple template-based system provides 100% reliability and competitive human evaluation scores, highlighting the limitations of current end-to-end data-driven approaches.
Introduction
Natural Language Generation (NLG) involves converting structured data, such as restaurant attributes, into fluent human-readable text. The E2E NLG Challenge was established to advance end-to-end data-driven methods that learn this mapping directly from data. However, these models are highly sensitive to the quality of training data. The authors observed that crowd-sourced datasets often contain "noise"—discrepancies where references omit key information or include extraneous details. This study investigates whether the complexity of neural models is justified or if a simple template-based system can offer superior performance in real-world scenarios.
Methods
The authors developed two distinct systems to compare these paradigms:
Model-D (Data-driven)
Model-D is a neural encoder-decoder architecture. Unlike standard models that treat input as a sequence, Model-D exploits the fixed structure of the input attributes. Each attribute is assigned a "positional ID" (e.g., name is always position 6), enabling a more efficient, non-sequential encoding process. The architecture incorporates an embedding matrix, a dense hidden layer for encoding, and a GRU-based decoder with an attention mechanism.
Model-T (Template-based)
Model-T is a deterministic system built in just a few hours. It utilizes a "master template" (e.g., [name] is a [eatType]...) and a set of hand-written rules to manage variations, such as phrasing numerical ratings or omitting missing attributes.
Dataset Analysis
The study utilizes the E2E NLG Dataset, containing over 50,000 instances in the restaurant domain. Each instance pairs a Meaning Representation (MR) with multiple human-written references. A manual analysis by the authors revealed that approximately 40% of the dataset contained discrepancies, such as omitted or modified content. This noise significantly impacts the learning process for neural models.
Key Results and Discussion
The evaluation revealed a stark contrast between automatic metrics and linguistic reliability.
The Metric Trap
Model-D outperformed the TGen baseline across all automatic metrics, achieving a BLEU score of 0.7128. However, these metrics reward models that closely mimic the training data. Because the training data was noisy, Model-D "learned" to drop content, failing to provide factual consistency.
Reliability vs. Diversity
In a manual audit of 100 samples, Model-T exhibited zero errors in grammar or content. Conversely, Model-D dropped information in 49 instances and produced 12 punctuation errors. While neural models offer more varied phrasing, they lack the predictability required for many real-world applications.
Human Evaluation
Despite lower automatic scores, Model-T was ranked in the second-best cluster for both quality and naturalness in the final challenge standings. This indicates that human evaluators prioritize grammatical correctness and factual accuracy over lexical variety.
Conclusion
The paper serves as a cautionary tale for the NLG community. While end-to-end neural architectures are powerful, they are currently limited by the quality of available training data. Future research should focus on reformulating loss functions to better handle noisy references. Ultimately, the transparency and reliability of symbolic methods like Model-T remain highly competitive, especially when data quality cannot be guaranteed.
Summary updated