ReportGPT: Human-in-the-loop Verifiable Table-to-Text Generation

Lucas Cecchi, Petr BabkinEMNLP 2024 · DOI

TL;DR

This paper introduces ReportGPT, a novel framework for generating verifiable text from tabular data with human oversight. It leverages a domain-specific language (DSL) to ensure factual accuracy and allows for real-time learning from user feedback, striking a balance between precision and insightful commentary.

What Problem Does This Paper Address?

The proliferation of large language models (LLMs) has led to a surge in data-to-text generation tools. However, a critical challenge remains: ensuring the accuracy and factuality of the generated text, especially when dealing with numerical data in tables. LLMs can "hallucinate" or omit information, leading to misleading content. Existing methods often struggle with numerical calculations or lack robust mechanisms for human verification. This paper addresses the need for a system that not only generates text from tables but also allows humans to easily verify its correctness and relevance, and learns from this feedback to improve over time.

Methods

ReportGPT employs a pipeline approach with several key modules:

  • Domain Specific Language (DSL): This is the core innovation. It's a simple, human-readable language designed for calculations over tabular data. The DSL acts as an intermediate representation, grounding the generated text and serving as a "proof mechanism" for verification. This approach limits the complexity of operations compared to full programming languages like Python, prioritizing readability and verifiability.
  • Program Generation: This module creates programs in the DSL based on an input table. The paper explores three approaches:
  • PCFG-based: Uses a Probabilistic Context-Free Grammar to generate programs. This method offers high precision but can be less insightful.
  • LLM-based: Leverages LLMs (specifically GPT-4) with carefully crafted prompts and few-shot examples to generate programs.
  • Chained LLM-based: Chains two LLM calls. The first generates questions about the table, and the second generates programs to answer those questions, aiming for more insightful commentary.
  • Program Execution: This module interprets and executes the generated DSL programs on the input table, producing results.
  • Surface Realization: This module translates the program and its execution result into a natural language sentence. This is also implemented using an LLM (GPT-4).
  • Human Feedback: This module is crucial for the "human-in-the-loop" aspect. Users review generated (commentary, program, result) tuples, accepting or rejecting them based on accuracy and relevance. This feedback is used to update the Program Generation and Surface Realization modules in real-time, improving the system's performance with just a few samples.

The paper also introduces a table linearization method that represents tables by their title, row headers, and column headers, reducing token count for LLM processing.

Datasets

The paper uses the HiTab dataset for evaluation. HiTab contains 3,700 complex tables from over 30 domains, including noise from missing cells, making it a realistic benchmark for real-world data.

Key Insights and Ideas

  • DSL as a Verifiability Enabler: The domain-specific language is key to making the generated text verifiable. By linking commentary directly to a DSL program that references table elements, users can quickly check the factual basis of the statements.
  • Trade-off Between Precision and Insightfulness: The experiments reveal a clear trade-off. PCFG-based generation leads to higher acceptance rates (precision) but less novel commentary. LLM-based approaches, especially the chained version, can produce more insightful text but may have lower acceptance rates.
  • Human Feedback as a Powerful Learning Signal: The system learns effectively from human feedback, requiring only a few samples to improve. This makes the human-in-the-loop process efficient and impactful.
  • Insightfulness is Complex: The study suggests that simply increasing the number of operations in a program doesn't necessarily lead to more insightful commentary, indicating that insightfulness is a more nuanced semantic quality.

Key Results

  • High Acceptance Rates: The PCFG-based configuration achieved the highest acceptance rates (up to 98% with titles) in Experiment 1, indicating high factual accuracy.
  • Insightfulness Ranking: The Chained LLM approach received the best average rank for insightfulness, suggesting it generates more descriptive and useful commentary.
  • LLM Configurations Improve with Few-Shot Learning: Acceptance rates and insightfulness generally improved for LLM-based configurations with 1-shot and 3-shot learning compared to zero-shot.
  • Titles Matter: Manually adding titles to tables significantly improved performance for all configurations.

Limitations and Future Work

  • Latency: The reliance on LLM API calls (GPT-4) for Program Generation and Surface Realization introduces latency. The paper addresses this with batching, but real-time interaction can still be slow.
  • Insightfulness Measurement: While the paper attempts to quantify insightfulness through user rankings, it acknowledges that this is a complex semantic quality and automating it based on proxy metrics is challenging.
  • Limited DSL Expressiveness: The DSL is intentionally limited for verifiability. More complex calculations might require extensions or different approaches.
  • Future Work: The authors suggest exploring hybrid configurations that combine the strengths of different approaches (e.g., PCFG for precision and LLMs for insightfulness) and further research into more robust methods for measuring and generating insightful commentary.

Summary updated

BibTeX
@inproceedings{cecchi2024reportgpt,
  author = {Lucas Cecchi and Petr Babkin},
  title = {{ReportGPT: Human-in-the-loop Verifiable Table-to-Text Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-industry.39},
}
Figure