Data-to-text Generation with Macro Planning

Ratish Puduppully, Mirella Lapata

TL;DR

This paper introduces a two-stage "plan-then-generate" framework for turning structured data (like sports statistics) into long-form, multi-paragraph text. By adding a macro planning stage that selects and orders high-level paragraph content before the actual writing begins, the authors significantly improve the factuality, coherence, and structure of AI-generated summaries.

What Problem Does This Paper Address?

Modern neural networks are surprisingly good at being "fluent"—they can generate text that sounds like a human wrote it. However, in the field of data-to-text generation (e.g., turning a box score into a sports article), they suffer from two major flaws:

  1. Hallucination: They often invent facts that aren't in the data.
  2. Poor Structure: They struggle to organize information logically across multiple paragraphs, often jumping between topics or repeating themselves.

While previous research focused on "micro planning" (deciding which individual data records to mention), this paper tackles macro planning: the high-level organization of a document. The goal is to give the model a "bird's-eye view" of the story it wants to tell before it starts worrying about individual word choices.

Methods

The authors propose a two-stage neural architecture that mimics the traditional pipeline of human writing:

  1. Macro Planning Stage:
    • The model first looks at the raw data and identifies "paragraph plans." These are clusters of entities (players/teams) and events (scoring plays) that typically appear together.
  2. It uses a Pointer Network to select which paragraph plans to include and in what order.
  3. Crucially, these plans are "contextualized"—the model considers how one paragraph relates to others using an attention mechanism before finalizing the sequence.
  4. Generation Stage:
    • This is a standard encoder-decoder model (with attention and a copy mechanism), but instead of reading the messy raw data table, it reads the macro plan produced in stage one.
  5. Because the "what to say" and "in what order" have already been decided, the generator can focus entirely on "how to say it" fluently.

To train this, the authors had to "reverse engineer" plans from existing human-written summaries by matching entities and events in the text back to the original data tables.

Datasets

The model was evaluated on two popular benchmarks for long-form data-to-text generation:

  • ROTOWIRE: Contains NBA basketball game statistics and human-written summaries (~5,000 instances).
  • MLB: A much larger dataset of Major League Baseball statistics and summaries (~25,000 instances). This dataset is more challenging due to its richer vocabulary and longer summaries (averaging over 500 words).

Key Insights and Ideas

  • Paragraphs as the Unit of Thought: The authors argue that paragraphs are the smallest domain where coherence and topic are truly defined. By planning at the paragraph level rather than the sentence or record level, the model maintains a better narrative flow.
  • Decoupling Logic from Language: By separating content selection (planning) from surface realization (writing), the model avoids the "bottleneck" where it tries to do too much at once, which often leads to hallucinations.
  • Latent Structure Induction: Since datasets don't come with "plans," the authors demonstrated that you can successfully induce a high-level structure by looking at lexical overlaps between data and text, effectively teaching the model the "latent" strategy humans use to write sports recaps.

Key Results

The "Macro" model outperformed several state-of-the-art baselines (including micro-planning and entity-centric models) across both datasets:

  • Factuality: On the MLB dataset, the Macro model achieved the highest Relation Generation (RG) precision (94.4%) among all neural models, meaning it was much less likely to lie about the data.
  • Coherence: In human evaluations, the Macro model was rated significantly higher for Coherence and Grammaticality than standard end-to-end models.
  • Content Selection: The model achieved a significant jump in Content Selection (CS) F-measure on ROTOWIRE (42.9 vs. the previous best of 40.5), indicating it is better at picking the "right" things to talk about.

Limitations and Future Work

Despite the improvements, the authors noted two main hurdles:

  • Attention Diffusion: In very long games (MLB summaries can be based on over 3,000 data tokens), the attention mechanism sometimes "blurs," leading the model to swap names (e.g., attributing a play to the wrong pitcher because their names appeared near each other in the plan).
  • Numeracy Issues: The model still doesn't truly "understand" numbers. For example, it might see a player shot 24% but change it to 30% because the language model "expects" a higher number in that context.
  • Future Directions: The authors suggest exploring noisy channel decomposition to better translate plans to text and developing specialized mechanisms for better numerical reasoning.

Summary updated

Abstract

Recent approaches to data-to-text generation have adopted the very successful encoder-decoder architecture or variants thereof. These models generate text which is fluent (but often imprecise) and perform quite poorly at selecting appropriate content and ordering it coherently. To overcome some of these issues, we propose a neural model with a macro planning stage followed by a generation stage reminiscent of traditional methods which embrace separate modules for planning and surface realization. Macro plans represent high level organization of important content such as entities, events and their interactions; they are learnt from data and given as input to the generator. Extensive experiments on two data-to-text benchmarks (RotoWire and MLB) show that our approach outperforms competitive baselines in terms of automatic and human evaluation.
BibTeX
@article{puduppully2021data,
  author = {Ratish Puduppully and Mirella Lapata},
  title = {{Data-to-text Generation with Macro Planning}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00381},
}

Loading related content...

Figure