TL;DR
This paper introduces a novel, efficient, and generalizable framework for Natural Language Querying (NLQ) on structured data, moving away from complex semantic parsing towards a pipeline of well-studied NLP tasks. Their approach breaks down NLQ into domain classification, multi-head slot extraction, and slot value disambiguation, making it more robust, interpretable, and easier to annotate for real-world applications.
What Problem Does This Paper Address?
The paper tackles the challenge of enabling users to query structured data (like databases) using natural language questions, a task that has become increasingly important with the rise of AI assistants and the need to access information locked in databases. Traditional approaches often rely on complex semantic parsing to translate natural language into formal query languages like SQL. However, these methods face significant hurdles: they require extensive, domain-specific annotated data, annotators need deep knowledge of database schemas and query languages, and models are brittle to schema changes. This makes them difficult and expensive to build, maintain, and scale in real-world commercial settings. The authors aim to provide a more practical, efficient, and adaptable solution.
Methods
The core of the proposed framework is a four-step pipeline:
- Domain Classification: Identifies which database table (or domain) is relevant to the user's query. This also helps detect out-of-scope queries.
- Multi-Head Slot/Entity Extraction: This is framed as a multi-head token classification task. It predicts:
- Type Head: The specific column/field name (e.g., "product name," "date").
- Role Head: The function of the slot (e.g.,
filter,sort_asc,aggregation). This is multi-label as a slot can have multiple roles. -
Function Head: The aggregation function (e.g.,
count,sum,avg). The authors use a DIET model (from Rasa) for this, trained with multi-task learning (MTL) to leverage positive transfer between domain and slot prediction. -
Slot Value Disambiguation (SVD): Raw slot values extracted from text (e.g., "last week," "amzn") need to be resolved into a format suitable for database queries. This is a crucial step, and the paper proposes a suite of methods depending on the slot type:
- Numeric slots (strict equality): Minimal preprocessing for exact matches (e.g., account numbers).
- Numeric slots (equality or ranges): Uses Duckling for normalization of dates, amounts, etc., and custom rules to handle variations.
- Non-contextual Textual SVD: Compares the raw span against a set of candidates using lexical similarity (fuzzy string matching) or semantic similarity (phrase embeddings).
-
Contextual Textual SVD: Uses the utterance context to disambiguate ambiguous terms (e.g., "apple" referring to a store vs. the company). An ensemble re-ranking approach is used to combine multiple SVD modules for improved accuracy.
-
Query Formulation: A deterministic algorithm translates the resolved slots, types, and roles into a database query. This step handles AND/OR conditions and ranges.
Datasets
The paper uses an internal dataset for training and evaluation, focusing on two large database tables in the retail and investment space, plus a third "other" domain for out-of-scope queries. They report statistics for domain classification and slot tagging on this dataset. No new benchmarks are introduced, but the work is evaluated on its practical applicability to real-world business use cases.
Key Insights and Ideas
- Decomposition is Key: Breaking down the complex NLQ problem into simpler, mature NLP tasks (classification, token tagging, disambiguation) makes the system more manageable, interpretable, and robust.
- MTL for Efficiency: Using Multi-Task Learning for domain classification and slot extraction significantly improves performance by leveraging shared representations.
- Flexible SVD: The multi-pronged approach to Slot Value Disambiguation (SVD) is critical. Recognizing that different slot types require different disambiguation strategies (numeric vs. textual, contextual vs. non-contextual) is a pragmatic insight.
- Interpretability and Debugging: The modular nature of the pipeline allows for individual component testing, improvement, and easier debugging when issues arise, which is paramount for production systems.
- Schema Agnosticism: The framework is designed to be largely agnostic to the specific database type (SQL/NoSQL) and can adapt to schema changes (like adding columns) without requiring full re-training of the NLU components.
Key Results
- Domain Classification: Achieved an F1 score of 89.64%.
- Slot Tagging: The overall pipeline achieved an average F1 score of 93.91% across various slot types.
- SVD Performance: The paper reports high F1 scores for SVD on different slot types, with product type SVD reaching 99.27% and amounts SVD achieving 97% accuracy.
- Practicality: The approach is demonstrated to be effective on an internal dataset and is currently deployed in production, handling queries involving filtering, sorting, and aggregation, with limited join capabilities.
Limitations and Future Work
- Complex Joins: The current framework struggles with complex joins, especially those involving different join types or columns with different names across tables. The authors suggest exploring decoder-based approaches for generating join sequences.
- Subquerying: The system does not currently support subqueries, which are notoriously difficult even for state-of-the-art semantic parsing models.
- Comparative Evaluation: A direct benchmark against existing text-to-SQL models on public datasets like Spider was not performed due to the infeasibility of re-annotating those datasets with their specific labels. However, they claim competitive performance on private datasets.
- Future Work: Extending the framework to handle more complex joins and subqueries are key areas for future development.
Summary updated