nBIIG: A Neural BI Insights Generation System for Table Reporting

Yotam Perlitz, Dafna Sheinwald, Noam Slonim, Michal Shmueli-Scheuer

TL;DR

This paper introduces nBIIG, a novel system that automatically generates natural language insights from tables, a crucial task in Business Intelligence (BI). By combining a robust analytics module with a neural surface realization model, nBIIG achieves significantly higher fluency than traditional template-based methods while maintaining strong faithfulness to the data, making BI reports more engaging and useful for analysts.

What Problem Does This Paper Address?

Business Intelligence (BI) tools are essential for organizations to make sense of their data. A key output of BI is the "table report," which summarizes findings in natural language. Manually creating these reports is time-consuming. Existing automated insight generation systems typically use a two-stage approach: first, performing statistical analyses on the table data to create a structured representation (like RDF), and second, using pre-defined templates to convert these representations into text. While this template-based method ensures high faithfulness (accuracy to the data), the generated insights often lack fluency and sound robotic, requiring analysts to heavily edit them. More recent end-to-end neural approaches can generate fluent text but often struggle with faithfulness, making them unsuitable for commercial applications. This paper addresses the gap of generating insights that are both highly fluent and faithful to the underlying data, aiming to create a practical and useful BI tool.

Methods

nBIIG employs a two-stage architecture, similar to traditional systems, but with a key neural innovation in the second stage.

  1. Analytics Module: This module performs a suite of 10 common BI analyses (e.g., MAX, MIN, SUM, AVERAGE, COMPARE, TREND) on the input table. The results of each analysis are converted into an RDF (Resource Description Framework) representation.
  2. Neural Surface Realization Module: This is the core novelty. Instead of using rigid templates, nBIIG uses a T5 (a transformer-based encoder-decoder model) to translate the RDF representations into natural language insights. This neural approach allows for more flexible and human-like sentence generation.

To train the T5 model, the authors used two approaches:

  • Extractive Approach: They collected 100K pairs of (RDF, sentence) from publicly available webpages where human experts wrote summaries for tables.
  • Data Augmentation: To address the scarcity of examples for more complex analysis types, they used GPT-NeoX-20B to generate synthetic textual insights for RDF representations that lacked human-annotated examples.

Additionally, a Recommender module combines the faithfulness score of an insight with user preference data to suggest relevant insights to the analyst.

Datasets

The paper mentions training data derived from:

  • Publicly accessible webpages: Specifically, data from statista.com was used, containing tables and human-written summaries. This yielded approximately 75K sentence/RDF matching pairs.
  • GPT-NeoX-20B: Used for data augmentation to generate synthetic insights for under-represented analysis types.

For evaluation, they sampled 66 tables and 529 insights.

Key Insights and Ideas

  • Neural Surface Realization is Key: Replacing deterministic templates with a neural model (T5) is the critical factor enabling nBIIG to achieve significantly higher fluency.
  • Bridging the Faithfulness-Fluency Gap: The paper demonstrates that a two-stage approach, when incorporating neural generation in the second stage, can successfully balance the often-conflicting goals of data faithfulness and textual fluency.
  • Data Augmentation for Robustness: The use of LLMs like GPT-NeoX-20B for data augmentation is crucial for improving the system's performance on less common or more complex analytical tasks, which are often underrepresented in manually curated datasets.
  • Human-in-the-Loop is Practical: The system is designed for practical use with an analyst. By providing confidence scores for faithfulness and allowing easy editing, nBIIG empowers users to refine insights, making the automation process collaborative rather than fully autonomous.

Key Results

  • Fluency Gain: Experts found nBIIG insights to be more than three times as fluent as those generated by a commercial template-based system. Specifically, 93% of nBIIG insights were found to be fluent, compared to only 23% for the template-based method.
  • Faithfulness Maintained: Despite the significant fluency improvement, nBIIG maintained a high level of faithfulness. On average, 92% of insights were found to be faithful to the table data across 10 supported analysis types.
  • Computational Efficiency: The system is designed to be computationally modest, with a single low-end GPU sufficient for maximal performance, making it practical for deployment.

Limitations and Future Work

  • Faithfulness is Not Perfect: While high, faithfulness is not 100%. The authors acknowledge that neural models are prone to errors and emphasize the importance of the human-in-the-loop approach for users to correct any inaccuracies.
  • Future Work: The authors intend to learn from user interactions to further improve the system's quality and explore more advanced ways to integrate user feedback. They also aim to expand the range of supported analyses and potentially explore end-to-end generation with improved faithfulness.

Summary updated

Abstract

We present nBIIG, a neural Business Intelligence (BI) Insights Generation system. Given a table, our system applies various analyses to create corresponding RDF representations, and then uses a neural model to generate fluent textual insights out of these representations. The generated insights can be used by an analyst, via a human-in-the-loop paradigm, to enhance the task of creating compelling table reports. The underlying generative neural model is trained over large and carefully distilled data, curated from multiple BI domains. Thus, the system can generate faithful and fluent insights over open-domain tables, making it practical and useful.
BibTeX
@inproceedings{perlitz2023nbiig,
  author = {Yotam Perlitz and Dafna Sheinwald and Noam Slonim and Michal Shmueli-Scheuer},
  title = {{nBIIG: A Neural BI Insights Generation System for Table Reporting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27082},
}
Figure