Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation

Albert Gatt, Emiel Krahmer

TL;DR

This paper provides a comprehensive survey of the state-of-the-art in Natural Language Generation (NLG), focusing on core tasks, applications, and evaluation methods. It highlights the significant shift towards data-driven approaches and the increasing integration of NLG with other AI fields, offering a valuable resource for researchers and practitioners.

What Problem Does This Paper Address?

The paper addresses the need for an updated overview of Natural Language Generation (NLG), a field that has undergone substantial changes over the past two decades. Specifically, it aims to synthesize recent advancements in data-driven methods, explore emerging applications (like image captioning and creative text generation), and critically examine the challenges and methodologies in evaluating NLG systems. The authors note that while NLG has been a part of AI and NLP for a long time, it has only recently begun to fully leverage modern machine learning techniques, necessitating a comprehensive survey of its current landscape.

Methods

The paper categorizes NLG approaches into three main architectural types:

  • Modular architectures: These systems feature distinct modules for specific tasks (e.g., content determination, text structuring, realization), often arranged in a pipeline.
  • Planning perspectives: This approach views text generation as a planning problem, where communicative goals are achieved through a sequence of actions, allowing for a more integrated perspective.
  • Integrated or global approaches: This dominant trend in NLG (and NLP generally) blurs task divisions, heavily relying on statistical learning to map non-linguistic inputs directly to outputs.

The survey also discusses the methodological orientation of these approaches, distinguishing between symbolic/knowledge-based methods and stochastic/data-driven methods. A significant theme is the shift from traditional rule-based systems to data-driven techniques, driven by the availability of large datasets and advancements in machine learning, including deep learning.

Datasets

The paper mentions several datasets used in NLG research, particularly in the context of image captioning and other data-driven applications. These include:

  • Flickr8k, Flickr30k, MS-COCO: Widely used datasets for image captioning, consisting of images paired with human-authored captions.
  • SBU1M Captioned Photo Dataset: Contains naturally-occurring captions from user-shared photographs.
  • Caltech UCSD Birds (CUB) dataset: A specialized dataset for bird descriptions.
  • WEATHERGOV corpus: Used for weather forecasting reports.
  • ROBOCUP dataset: Used for sports commentary generation.
  • BAGEL dataset: Used for dialogue act generation.

The paper also notes the importance of aligned data (input paired with corresponding text) for many statistical NLG systems, but highlights that newer deep learning approaches are reducing the reliance on strict alignment.

Key Insights and Ideas

  • The "Generation Gap": A persistent challenge where early strategic decisions in the NLG pipeline can have unforeseen negative consequences on later tactical components, leading to mismatches between intent and output.
  • Data-Driven Dominance: The field has seen a dramatic shift from hand-crafted, rule-based systems to statistical and data-driven approaches, especially with the rise of deep learning. This trend is making NLG systems more adaptable and scalable.
  • Blurring Task Boundaries: Modern approaches, particularly integrated and planning-based ones, are increasingly blurring the lines between traditional NLG tasks like content determination, text structuring, and realization.
  • The Importance of Evaluation: Evaluating NLG systems is complex due to variable inputs, multiple possible outputs, and the subjective nature of language quality. The paper emphasizes the need for diverse evaluation methods, combining intrinsic and extrinsic measures, and acknowledges the ongoing research into the correlation between automatic metrics and human judgments.
  • NLG as a Testbed for AI Theories: The paper suggests that NLG systems can serve as valuable testbeds for theories in areas like pragmatics, cognitive science, and computational creativity, offering a reciprocal benefit to both fields.

Key Results

The paper is a survey and does not present novel experimental results. Instead, it synthesizes findings from numerous research papers. Key themes that emerge from the surveyed research include:

  • Improved Fluency and Coherence: Data-driven and integrated approaches have led to more fluent and coherent generated text compared to earlier modular systems.
  • Scalability and Adaptability: The shift to data-driven methods has enabled NLG systems to scale to larger domains and adapt to specific applications more effectively.
  • Emergence of New Applications: Significant progress has been made in areas like image captioning, stylistic variation, and creative text generation, demonstrating NLG's expanding capabilities beyond factual reporting.
  • Challenges in Evaluation: Despite advancements, evaluating NLG output remains a significant challenge, with ongoing research into reliable and consistent metrics.

Limitations and Future Work

Limitations: * Scalability: While data-driven approaches offer scalability, applying them to domains with heterogeneous data and generating long texts remains a challenge. * Efficiency and Robustness: Some advanced approaches, particularly those rooted in AI planning paradigms, can be computationally expensive. * Data Acquisition: Obtaining sufficient and appropriate data for training statistical models is still a bottleneck for some NLG tasks. * Evaluation Consistency: The lack of consistent correlation between automatic metrics and human judgments poses a challenge for reliable system comparison.

Future Work: * Cross-disciplinary Collaboration: The authors strongly advocate for increased collaboration between NLG researchers and those in related fields like computer vision, computational creativity, and cognitive science. * Social Media and Situated Generation: Future research should focus on generating language for social media contexts and in situated, interactive environments. * Deep Learning Architectures: Continued exploration of deep learning models, especially those that learn domain-invariant representations and facilitate transfer learning, is crucial. * Understanding "Quality": Further research is needed to better understand and quantify the dimensions of quality in NLG, such as what constitutes "fluency" or "adequacy" and how consistently these are judged. * Data-Text Alignment: Developing techniques for less precise data-text alignment will be important for broader applicability.

Summary updated

Abstract

This paper surveys the current state of the art in Natural Language Generation (NLG), defined as the task of generating text or speech from non-linguistic input. A survey of NLG is timely in view of the changes that the field has undergone over the past two decades, especially in relation to new (usually data-driven) methods, as well as new applications of NLG technology. This survey therefore aims to (a) give an up-to-date synthesis of research on the core tasks in NLG and the architectures adopted in which such tasks are organised; (b) highlight a number of recent research topics that have arisen partly as a result of growing synergies between NLG and other areas of artificial intelligence; (c) draw attention to the challenges in NLG evaluation, relating them to similar challenges faced in other areas of NLP, with an emphasis on different evaluation methods and the relationships between them.
BibTeX
@article{gatt2018survey,
  author = {Albert Gatt and Emiel Krahmer},
  title = {{Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation}},
  journal = {J. Artif. Intell. Res.},
  year = {2018},
  url = {https://openreview.net/forum?id=Bs2ZXpvLzl},
}
Figure