The Natural Language Pipeline, Neural Text Generation and Explainability

Juliette Faille, Albert Gatt, Claire GardentNL4XAI@INGL 2020 · OpenReview

TL;DR

This paper surveys recent neural data-to-text generation models that break down the end-to-end process into traditional Natural Language Generation (NLG) pipeline sub-tasks. The authors explore how these modular approaches can enhance model explainability, moving beyond "black box" end-to-end systems.

What Problem Does This Paper Address?

Modern neural networks, particularly end-to-end encoder-decoder models, excel at generating fluent text from structured data. However, these models often act as "black boxes," making it difficult to understand why they produce certain outputs or to diagnose and fix issues like factual inaccuracies (hallucinations) or omissions. Traditional NLG systems, in contrast, were built with distinct modules for planning what to say (macroplanning) and how to say it (microplanning and surface realization). This paper investigates how recent neural models are re-integrating these traditional modular structures to improve the explainability of data-to-text generation.

Methods

The paper surveys existing research, categorizing approaches based on which traditional NLG sub-tasks they model:

  • Macroplanning: This involves selecting and organizing content. Some models perform explicit "content determination" (e.g., by masking input tokens), while others rely on template selection or hierarchical planning.
  • Document Structuring: This organizes the selected content into sentences and paragraphs. Approaches include using latent variables (like learned templates), explicit content structuring (e.g., ordered trees), input structure encoding (leveraging the input data's inherent structure), and dedicated attention mechanisms.
  • Microplanning: This focuses on the linguistic realization of the content. This includes:
  • Referring Expression Generation (REG): Generating phrases to refer to entities. Some models handle this as a post-processing step, while others integrate it more deeply.
  • Lexicalization: Mapping input symbols to words. This is often handled by the decoder, with copy mechanisms allowing direct copying from the input.
  • Surface Realization: Creating syntactically correct text. This is often implicitly handled by the decoder's language model or by templates.

The core idea is that by explicitly modeling these sub-tasks, either through intermediate structures or latent variables, researchers can gain insights into the generation process.

Datasets

The paper does not introduce new datasets. It surveys existing work that utilizes various data formats as input, including:

  • RDF triples
  • Tables
  • Text (for headline generation)
  • Meaning Representations (MR)

Specific datasets are not detailed for each surveyed paper, but the input types indicate the domains these models are applied to.

Key Insights and Ideas

  • Modularization for Explainability: The central thesis is that breaking down complex end-to-end models into more interpretable sub-modules, mirroring traditional NLG pipelines, is a promising path to explainability.
  • Latent Variables as Explanations: Models that use latent variables (e.g., learned templates) to mediate between input and output can offer explanations by showing how input elements map to generated text fragments.
  • Explicit Structures Aid Analysis: Approaches that output explicit content plans, sentence templates, or referring expressions allow for direct examination and qualitative analysis of the generation process.
  • Template Learning: Neural parameterizations of templates can learn to structure content and even influence lexicalization, providing a form of explainable planning.
  • Trade-offs in Approaches: Explicit pipeline models offer direct interpretability but may require more labeled data. Latent variable models are more domain-agnostic but their internal workings can still be complex.

Key Results

The paper is a survey, not an empirical study, so it doesn't present new quantitative results. Instead, it highlights the potential for improved explainability offered by different modeling choices. For instance:

  • Latent variable approaches (like those in Wiseman et al., 2018) intuitively explain model behavior by showing how input licenses output.
  • Explicitly generated content plans (e.g., in Castro Ferreira et al., 2019) allow for detailed qualitative analysis of sub-tasks.
  • The "switch variable" in lexicalization (See et al., 2017) can indicate how much the model copies versus generates words, offering a glimpse into lexicalization choices.

Limitations and Future Work

  • Limited Scope: The authors acknowledge that their survey is not exhaustive due to space limitations.
  • Focus on Data-to-Text: The survey primarily focuses on generation from data and meaning representations, with a brief mention of text-to-text generation for content selection.
  • Future Directions:
    • Investigating whether domain-agnostic, linguistically inspired intermediate structures (like meaning representations) can improve both explainability and performance.
  • Developing common evaluation benchmarks and metrics specifically for analyzing the explainability of neural NLG models across different sub-tasks.
  • Exploring how methods for modeling the interaction between content planning and microplanning can be extended to improve the explainability of individual microplanning sub-tasks (lexicalization, aggregation, etc.).

Summary updated

Abstract

End-to-end encoder-decoder approaches to data-to-text generation are often black boxes whose predictions are difficult to explain. Breaking up the end-to-end model into sub-modules is a natural way to address this problem. The traditional pre-neural Natural Language Generation (NLG) pipeline provides a framework for breaking up the end-to-end encoder-decoder. We survey recent papers that integrate traditional NLG submodules in neural approaches and analyse their explainability. Our survey is a first step towards building explainable neural NLG models.
BibTeX
@inproceedings{faille2020natural,
  author = {Juliette Faille and Albert Gatt and Claire Gardent},
  title = {{The Natural Language Pipeline, Neural Text Generation and Explainability}},
  booktitle = {NL4XAI@INGL},
  year = {2020},
  url = {https://openreview.net/forum?id=WrQy9nv6Le},
}
Figure