Introduction
Natural language interfaces for data analysis often struggle with the inherent vagueness of human communication. When a user asks a system to "analyze summer sales in Europe," they rarely provide the precise definitions required for code execution—what months constitute "summer"? Which countries are in "Europe"? Traditionally, research has treated this underspecification as a technical hurdle to be overcome by identifying a single "correct" latent intent.
However, this paper argues that ambiguity is not a bug, but a fundamental feature of cooperative interaction. In real-world "open-domain" settings, users often lack the structural knowledge of the underlying database necessary to be perfectly precise. Current benchmarks for Text-to-SQL and Tabular Question Answering (QA) frequently bypass this reality by providing "data-privileged" queries that leak schema information, creating an artificial environment that fails to test a system's true interpretive capabilities.
The Lens of Cooperative Interaction
To better understand how systems should handle ambiguity, the authors propose a framework based on Gricean communicative principles. Queries are categorized into three distinct types based on their level of specification:
- Unambiguous Queries: These are fully specified; the system needs no additional inference to execute the task.
- Cooperative (Ambiguous) Queries: These are underspecified but resolvable through "reasonable inference," such as common sense or domain-specific conventions.
- Uncooperative Queries: These are so vague that they cannot be resolved without further clarification from the user.
This framework highlights that underspecification is often intentional. Users delegate specific choices (e.g., choosing between mean or median for "average") to the system, granting it agency to find the most appropriate interpretation.
Auditing Existing Benchmarks
The authors conducted a meta-analysis of 15 prominent tabular benchmarks, including Text-to-SQL datasets like Spider and BIRD, and complex analysis tasks like DA-Code. Using LLM-based classifiers (validated against human experts with a Cohen’s $\kappa$ of 0.866), they audited these datasets for:
- Data-Independence: Whether the query relies on "privileged" knowledge of the table, such as exact column headers.
- Specification Levels: How clearly the entity, time frame, domain, intent, and methodology are defined.
Results: The Prevalence of Data-Privilege
The audit revealed a significant disconnect between benchmark design and real-world usage. Most current datasets are "saturated" with data-privileged queries. For instance, a query like "List the first_name of users" assumes the user already knows the schema, which contradicts the premise of an open-domain natural language interface.
Furthermore, truly unambiguous queries are a small minority across almost all datasets. This means systems are constantly making interpretive "guesses" that are then judged against a single "gold" answer. This conflates execution accuracy (writing correct code) with interpretation capability (making human-aligned choices), leading to flawed evaluations of system intelligence.
Discussion and Future Directions
The findings suggest that we need to rethink how we evaluate data analysis systems. Ambiguity should be embraced as a space for cooperative dialogue rather than a problem to be eliminated in a single turn.
Key recommendations for future research include:
- Iterative Benchmarks: Developing datasets that allow for multiple valid "grounding paths" and multi-turn refinement.
- System Transparency: Designing systems that disclose their interpretive choices (e.g., "I've defined 'summer' as June-August; is that correct?").
- Robustness to Vagueness: Testing whether systems can identify "uncooperative" queries and ask for clarification instead of hallucinating a specific intent.
By moving away from data-privileged "scaffolding," we can begin to build systems that truly understand the nuances of human intent in data analysis.
Summary updated