Semantically Aligned Question and Code Generation for Automated Insight Generation

Ananya Singha, Bhavya Chopra, Anirudh Khatry, Sumit Gulwani, Austin Z. Henley, Vu Le, Chris Parnin, Mukul Singh, Gust Verbruggen

Abstract

Automated data exploration tools often struggle with "semantic misalignment," where generated code fails to accurately answer the natural language question posed. This paper presents a novel pipeline for automated insight generation that simultaneously produces diverse question-code pairs and utilizes an efficient, embedding-based classifier to ensure semantic alignment. Our approach achieves performance comparable to GPT-4 in alignment detection while being significantly more cost-effective and faster, offering a scalable solution for enhancing exploratory data analysis (EDA).

Introduction

Exploratory Data Analysis (EDA) is a critical yet time-consuming phase of the data science workflow. While Large Language Models (LLMs) have shown promise as "Copilots" by suggesting questions and generating corresponding code, they frequently suffer from semantic misalignment. This occurs when the "insight" (the question) and the "action" (the code) are out of sync—for instance, a model might ask for a "winning streak" but provide code that merely counts total wins. Such discrepancies can lead users to false conclusions. This paper introduces a robust pipeline designed to bridge this gap through joint generation and high-efficiency verification.

Methodology

The proposed system for Automated Insight Generation follows a multi-stage process designed for both diversity and accuracy:

1. Table Analysis and Contextual Prompting

The system begins by analyzing table metadata, including data types, cardinality, and missing values. A decision tree is employed to identify columns most likely to yield meaningful insights through grouping or aggregation. This metadata forms the basis of a context-rich prompt for the LLM.

2. Joint Generation (Q+C)

Unlike traditional methods that generate a question and then translate it into code, this pipeline instructs the model to generate both the Question and Code (Q+C) in a single pass. This "Joint Generation" approach has been found to produce more diverse results, preventing the model from becoming repetitive.

3. Candidate Filtering and Semantic Alignment

After generation, the system discards code that fails to execute or contains syntax errors. The remaining pairs are passed to the core technical innovation: a Semantic Alignment Classifier. This is a lightweight, fully connected neural network that takes the embeddings of the question and the code, concatenates them, and predicts whether they are aligned. By using vector representations rather than prompting a large model for verification, the system remains highly performant and inexpensive.

Experimental Datasets

The researchers utilized three primary data sources to train and validate the system:

  • Open-WikiTable: A subset of 430 tables from a larger corpus of 24,680 tables.
  • Human-Annotated Alignment Dataset: 309 (Question, Code) pairs labeled by expert data scientists as "aligned" or "misaligned."
  • Synthetic Misalignment Dataset: 1,552 pairs created by swapping code snippets between different questions to provide a robust training set for the classifier.

Key Insights and Results

Performance and Accuracy

The embedding-based classifier reached an accuracy of 85.1%, outperforming GPT-3.5 (73.7%) and GPT-4 (83.5%) in identifying misaligned pairs. It achieved a 90% F1 score, demonstrating that a specialized, smaller model can effectively verify the outputs of much larger generative models.

Efficiency and Cost

The system's efficiency is a major highlight. The embedding-based classifier is 728 times more cost-effective than using GPT-4 for alignment tasks. Furthermore, the Joint Generation method proved to be 37 times cheaper and 6 times faster than the traditional two-step "text-to-code" translation process.

User Experience and Diversity

In a study with five participants, 79% of users agreed the generated insights enhanced their understanding of the data, and 77% reported improved productivity. Interestingly, the study found that "zero-shot" prompting led to higher diversity than "few-shot" prompting; when given examples, the LLM tended to mimic the structure of those examples rather than exploring the unique properties of the dataset at hand.

Limitations and Future Work

While the pipeline is highly effective at ensuring technical accuracy, it currently lacks a sophisticated mechanism for ranking the "interestingness" of insights. Some generated pairs, while semantically aligned, were trivial (e.g., "How many rows are in this table?"). Additionally, the classifier's performance dropped slightly when moving from human-annotated to purely machine-generated data, indicating that LLM-generated errors have distinct characteristics. Future work will focus on ranking algorithms for insight relevance and expanding the system beyond English and Python/Pandas.

Conclusion

By shifting the focus from simple generation to semantically aligned joint generation and efficient verification, this research provides a scalable path forward for automated data exploration. The use of lightweight embedding-based classifiers proves that high-quality verification does not require the computational overhead of the largest LLMs, making sophisticated EDA tools more accessible and reliable.

Summary updated

BibTeX
@inproceedings{singha2024semantically,
  author = {Ananya Singha and Bhavya Chopra and Anirudh Khatry and Sumit Gulwani and Austin Z. Henley and Vu Le and Chris Parnin and Mukul Singh and Gust Verbruggen},
  title = {{Semantically Aligned Question and Code Generation for Automated Insight Generation}},
  booktitle = {LLM4CODE@ICSE},
  year = {2024},
  url = {https://openreview.net/forum?id=Q4FKo61jzH},
}
Figure