Are We Asking the Right Questions? On Ambiguity in Natural Language Queries for Tabular Data Analysis

Daniel Gomm, Cornelius Wolff, Madelon Hulsebos

Introduction

Natural language interfaces for data analysis often struggle with the inherent vagueness of human communication. When a user asks a system to "analyze summer sales in Europe," they rarely provide the precise definitions required for code execution—what months constitute "summer"? Which countries are in "Europe"? Traditionally, research has treated this underspecification as a technical hurdle to be overcome by identifying a single "correct" latent intent.

However, this paper argues that ambiguity is not a bug, but a fundamental feature of cooperative interaction. In real-world "open-domain" settings, users often lack the structural knowledge of the underlying database necessary to be perfectly precise. Current benchmarks for Text-to-SQL and Tabular Question Answering (QA) frequently bypass this reality by providing "data-privileged" queries that leak schema information, creating an artificial environment that fails to test a system's true interpretive capabilities.

The Lens of Cooperative Interaction

To better understand how systems should handle ambiguity, the authors propose a framework based on Gricean communicative principles. Queries are categorized into three distinct types based on their level of specification:

  1. Unambiguous Queries: These are fully specified; the system needs no additional inference to execute the task.
  2. Cooperative (Ambiguous) Queries: These are underspecified but resolvable through "reasonable inference," such as common sense or domain-specific conventions.
  3. Uncooperative Queries: These are so vague that they cannot be resolved without further clarification from the user.
The Lens of Cooperative Interaction: Categorizing queries and their mapping to actionable interpretations.

This framework highlights that underspecification is often intentional. Users delegate specific choices (e.g., choosing between mean or median for "average") to the system, granting it agency to find the most appropriate interpretation.

Auditing Existing Benchmarks

The authors conducted a meta-analysis of 15 prominent tabular benchmarks, including Text-to-SQL datasets like Spider and BIRD, and complex analysis tasks like DA-Code. Using LLM-based classifiers (validated against human experts with a Cohen’s $\kappa$ of 0.866), they audited these datasets for:

  • Data-Independence: Whether the query relies on "privileged" knowledge of the table, such as exact column headers.
  • Specification Levels: How clearly the entity, time frame, domain, intent, and methodology are defined.

Results: The Prevalence of Data-Privilege

The audit revealed a significant disconnect between benchmark design and real-world usage. Most current datasets are "saturated" with data-privileged queries. For instance, a query like "List the first_name of users" assumes the user already knows the schema, which contradicts the premise of an open-domain natural language interface.

Benchmark Analysis: (Left) Prevalence of data-privileged queries and specific reference types. (Right) Distribution of query specification levels across 15 major datasets.

Furthermore, truly unambiguous queries are a small minority across almost all datasets. This means systems are constantly making interpretive "guesses" that are then judged against a single "gold" answer. This conflates execution accuracy (writing correct code) with interpretation capability (making human-aligned choices), leading to flawed evaluations of system intelligence.

Discussion and Future Directions

The findings suggest that we need to rethink how we evaluate data analysis systems. Ambiguity should be embraced as a space for cooperative dialogue rather than a problem to be eliminated in a single turn.

Key recommendations for future research include:

  • Iterative Benchmarks: Developing datasets that allow for multiple valid "grounding paths" and multi-turn refinement.
  • System Transparency: Designing systems that disclose their interpretive choices (e.g., "I've defined 'summer' as June-August; is that correct?").
  • Robustness to Vagueness: Testing whether systems can identify "uncooperative" queries and ask for clarification instead of hallucinating a specific intent.

By moving away from data-privileged "scaffolding," we can begin to build systems that truly understand the nuances of human intent in data analysis.

Summary updated

BibTeX
@inproceedings{gomm2025are,
  author = {Daniel Gomm and Cornelius Wolff and Madelon Hulsebos},
  title = {{Are We Asking the Right Questions? On Ambiguity in Natural Language Queries for Tabular Data Analysis}},
  booktitle = {AITD @ EurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=Mm1CGvmm9Z},
}
Figure