Satyrn: A Platform for Analytics Augmented Generation

Marko Sterbentz, Cameron Barrie, Shubham Shahi, Abhratanu Dutta, Donna Hooshmand, Harper Pack, Kristian J. HammondEMNLP 2024 · DOI

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation. However, they remain "unreliable bookkeepers" when tasked with complex reasoning over structured data. While Retrieval Augmented Generation (RAG) effectively grounds models in textual corpora, it often fails when answers must be calculated rather than retrieved. For instance, answering "What is the average wildfire size in Oregon compared to the national average?" requires precise multi-step computation across database tables.

Existing approaches, such as OpenAI’s Code Interpreter, rely on the LLM to generate and execute its own code. This method is inherently non-deterministic and prone to "reasoning gaps," where the model may produce buggy SQL or fail to execute all necessary analytical steps. To address these challenges, we present SATYRN, a neurosymbolic platform that introduces Analytics Augmented Generation (AAG). SATYRN separates the analysis (logic and math) from the generation (prose), achieving significantly higher factual accuracy than state-of-the-art neural-only baselines.

System Architecture

The SATYRN architecture follows a "plan-then-generate" workflow designed to ground LLMs in verified facts derived from structured databases.

The SATYRN workflow: from input report requirements to deterministic analysis and final LLM-generated prose.

The system comprises four primary components:

  1. SATYRN Rings: A lightweight semantic layer that maps raw database tables to "Entities" and "Attributes." This allows the system to understand relationships, such as joins, without requiring the LLM to interpret the database schema dynamically.
  2. Report Blueprints: Templates for specific report categories (e.g., Ranking Reports) that define the informational requirements and the necessary analytical plans.
  3. Structured Question Representation (SQR): A domain-agnostic, graph-based language for defining analytic operations. SQR allows for precise definitions of operations like average, groupby, and sort without the fragility of raw SQL generation.
Example of Structured Question Representation (SQR) and its corresponding computational graph.
  1. The Analysis Engine: This deterministic symbolic engine converts SQR plans into SQL, executes them against the database, and populates natural language "statement templates" with the results.

By the time the LLM receives the data, it is no longer looking at raw tables but at a list of verified, natural language factual statements (e.g., "Oregon is ranked 4th in wildfire size"). The LLM's role is strictly limited to weaving these facts into a coherent, fluent narrative.

Experimental Evaluation

SATYRN was evaluated across eight diverse domains, including healthcare (MIMIC-IV), environmental sustainability (wildfire data), and socioeconomic census data. The study compared SATYRN against unaugmented GPT-4 and GPT-4 with Code Interpreter across 200 generated reports containing over 3,200 individual claims.

Key Results

  • Factual Accuracy: SATYRN achieved 86% factual accuracy, significantly outperforming GPT-4 with Code Interpreter (57%) and unaugmented GPT-4 (48%).
  • Model Efficiency: Remarkably, a small 7-billion parameter model (Mistral-7B) using the SATYRN framework outperformed GPT-4 using Code Interpreter. This suggests that when the heavy lifting of calculation is handled by a symbolic engine, smaller models are highly effective at the prose generation task.
  • Textualization vs. Raw Data: Ablation studies showed that providing the LLM with natural language facts rather than raw data tables (SATYRN-Table) led to much higher performance, indicating that LLMs are better at translating facts into prose than interpreting raw data structures.

Discussion and Future Work

The success of SATYRN highlights a critical insight: symbolic engines are superior to neural models for mathematical and logical operations. By using natural language as an intermediate representation between the symbolic engine and the LLM, SATYRN eliminates the non-determinism that leads to hallucinations in traditional LLM-based data analysis tools.

Current limitations include the requirement for structured JSON inputs and pre-defined report blueprints. Future work will focus on enabling natural language queries as the primary interface and implementing automated validation loops to ensure the final prose perfectly aligns with the source facts provided by the Analysis Engine.

Conclusion

SATYRN demonstrates that Analytics Augmented Generation is a robust path forward for grounding LLMs in structured data. By offloading computation to a deterministic symbolic engine, SATYRN provides a scalable, accurate, and efficient platform for generating data-driven reports across a wide variety of domains.

Summary updated

BibTeX
@inproceedings{sterbentz2024satyrn,
  author = {Marko Sterbentz and Cameron Barrie and Shubham Shahi and Abhratanu Dutta and Donna Hooshmand and Harper Pack and Kristian J. Hammond},
  title = {{Satyrn: A Platform for Analytics Augmented Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.365},
}
Figure