Conversing with databases: Practical Natural Language Querying

Denis Kochedykov, Fenglin Yin, Sreevidya KhatravathEMNLP 2023

TL;DR

This paper introduces DataQue, a hybrid Natural Language Querying (NLQ) system designed for conversational database interactions in enterprise settings. It tackles the challenge of translating complex, jargon-filled user questions into precise SQL queries by combining rule-based components with pre-trained language models, achieving 88% exact match accuracy.

What Problem Does This Paper Address?

Decision-makers in companies often struggle to access and analyze the vast amounts of data stored in relational databases. Traditional methods like static reports or complex SQL queries are inaccessible to non-technical users. While Natural Language Querying (NLQ) systems offer a promising solution by allowing users to ask questions in plain English, existing solutions often fall short. They struggle with the nuances of real-world business queries, which are frequently compressed, contain jargon, abbreviations, implied conditions, and custom calculations. Furthermore, these systems often lack the deterministic precision required for business-critical decisions, and they can be difficult to customize with domain-specific logic.

Methods

DataQue employs a hybrid framework that mixes rule-based components with model-based approaches. The core of the system is a pipeline consisting of several key components:

  • Intent Classifier: Distinguishes between conversational intents (like greetings) and actual data questions.
  • Exact Pattern Matching: Handles very simple, common queries with predefined SQL translations for guaranteed results.
  • NER (Named Entities Recognition) & NED (Named Entities Disambiguation): Identifies and resolves entities (like trading desk names, dates, numerical columns) within the user's query, linking them to database schema or specific values. This is crucial for understanding the query's intent.
  • Table Classifier: Determines which database table(s) the query refers to.
  • AST (Abstract Syntax Tree) Constructor: Generates an initial, structured representation of the query.
  • AST Expansion Component: This is a key innovation. It uses a rule-based system to inject implied conditions, default values, and custom business logic into the AST, addressing the complexity of real-world queries.
  • OOD (Out-Of-Domain) Classifier: Identifies questions that are outside the system's scope to prevent incorrect data retrieval.
  • Follow-on Classifier & DST (Dialog State Tracker): Manage conversational context, allowing users to refine previous queries or drill down into data.

The system leverages pre-trained transformer models for tasks like NER but relies heavily on custom-built rule-based components for AST expansion and other logic, aiming for controllability and precision.

Datasets

The paper highlights the challenge of domain-specific language and the lack of suitable public datasets. Therefore, DataQue's training and testing relied on internally collected data:

  • Synthetic questions: Approximately 10 template question structures were used to generate around 20,000 synthetic questions.
  • Annotated questions for NER/NED training: Around 300 example questions were manually annotated with entities and their resolutions.
  • Annotated questions for NER/NED testing: A similar set of approximately 200 questions was used for testing.
  • Examples of non-business utterances: Around 200 examples of greetings, affirmations, etc., were collected.
  • Production run questions: Approximately 300 business questions were sampled from the initial period of production use with real users.

Key Insights and Ideas

  • The "jargon gap" is real: Real business users ask questions that are far from "nice" textbook examples, filled with compressed language, jargon, and implied logic.
  • Hybrid approach is key: A purely model-based approach struggles with the deterministic precision and customizability needed for enterprise NLQ. Combining rule-based logic with ML models offers a practical balance.
  • AST as a flexible intermediate: Using an Abstract Syntax Tree (AST) provides a structured representation that can be manipulated by both model-based parsing and rule-based expansion, making it ideal for incorporating complex business logic.
  • Controllability vs. Recall: The paper acknowledges the inherent trade-off between rule-based systems (high precision, low recall) and statistical models (lower precision, higher recall). DataQue prioritizes precision and controllability for business-critical applications.
  • Explainability for trust: To mitigate the risk of users unknowingly receiving incorrect data, the system provides a "human-in-the-loop" path to show users the parsing results, allowing them to identify potential errors.

Key Results

  • Overall System Accuracy: DataQue achieved an 88% Exact Match (EM) score on production data, meaning the system returned the precise data expected by the user.
  • Component Accuracy:
    • NER: Weighted average F1 score of 97% across all entity types.
  • NED: Weighted average F1 score of 96% across all entity types.
  • Latency: The system achieves a latency of less than one second, which is significantly faster than potential LLM-based solutions.
  • Qualitative Findings: The system successfully handles complex implied conditions, jargon, and custom calculations, as demonstrated by its production deployment.

Limitations and Future Work

  • Limitations:
    • Rule Maintenance: The rule-based components require ongoing maintenance and updates, which can be effort-intensive for the development team.
  • Robustness to Variations: The system is less robust to variations in questions that require deep reasoning or common sense knowledge, and it may default to error messages for complex conversational scenarios outside simple question-follow-on patterns.
  • Future Work:
    • Larger Datasets: Collecting more user queries from production usage to create larger training and testing datasets for components.
  • Leveraging LLMs: Investigating how to integrate Large Language Models (LLMs) into the pipeline. While LLMs excel at NLU and code generation, challenges remain in achieving the required precision and managing latency. The authors envision a hybrid solution combining their current pipeline with LLM components.

Summary updated

BibTeX
@inproceedings{kochedykov2023conversing,
  author = {Denis Kochedykov and Fenglin Yin and Sreevidya Khatravath},
  title = {{Conversing with databases: Practical Natural Language Querying}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure