Tackling Temporal Questions in Natural Language Interface to Databases

Ngoc Phuoc An Vo, Octavian Popescu, Irene Manotas, Vadim SheininEMNLP 2022

TL;DR

This paper tackles the challenge of enabling natural language interfaces to databases (NLIDBs) to understand and answer temporal questions. The authors introduce a new dataset, TempQ4NLIDB, specifically designed for temporal questions, and evaluate its effectiveness in improving the performance of existing state-of-the-art NLIDB models, showing significant gains.

What Problem Does This Paper Address?

Natural Language Interfaces to Databases (NLIDBs) aim to bridge the gap between human language and structured data in databases. While significant progress has been made, handling temporal aspects within these queries remains a major hurdle. Temporal information is ubiquitous in real-world databases, reflecting the dynamic nature of data. However, challenges arise from the complex ways temporal information is represented and associated with data, as well as the inherent ambiguity of natural language expressions for time. Existing NLIDB benchmarks and models often under-attend to this crucial dimension, leading to systems that struggle to accurately retrieve time-sensitive information.

Methods

The authors propose a new dataset, TempQ4NLIDB, to address the lack of resources for temporal questions in NLIDBs. This dataset focuses on "temporal condition questions," which are considered the most practical and challenging type. They also introduce three variants of temporal questions to facilitate model learning:

  • Original Temporal Questions: These retain the original, unnormalized temporal expressions, posing the greatest challenge for models to map to database date values.
  • Full-DA Questions: Temporal questions are pre-processed with data annotation, and normalized date values are appended to the question. This variant is inspired by how some models learn values.
  • Partial-DA Questions: Similar to Full-DA, but normalized date values are not explicitly appended to the question, easing the mapping process.

The paper also details an annotation guideline for temporal questions, including rules for mapping temporal operators to SQL and generating natural language questions from SQL queries. They employ an "ad-hoc date annotator" to normalize temporal expressions.

Datasets

  • Spider: Used for pre-training and initial evaluation of models. The authors analyze temporal questions within the Spider dataset.
  • TempQ4NLIDB: A newly introduced dataset specifically designed for temporal questions in NLIDBs. It includes two synthetic databases (Human Resource and Warehouse) and temporal condition questions with corresponding SQL queries. The dataset also includes non-temporal questions for comparative evaluation.

Key Insights and Ideas

  • Temporal questions are a critical, yet under-addressed, area in NLIDB. The inherent complexity of temporal expressions and their representation in databases makes them a significant challenge for current systems.
  • Existing models trained solely on general NLIDB datasets (like Spider) perform poorly on temporal questions. This highlights the need for specialized datasets and training strategies.
  • The TempQ4NLIDB dataset, with its focus on temporal conditions and data variants, significantly improves model performance. This demonstrates the value of targeted datasets for specific linguistic phenomena.
  • Data augmentation techniques, such as normalizing temporal expressions (Partial-DA and Full-DA variants), are crucial for helping models learn to handle temporal queries effectively. This suggests that pre-processing and providing explicit temporal cues can bridge the gap between natural language and database logic.
  • Handling both temporal and non-temporal questions simultaneously increases complexity for models. This indicates that models need to be robust to a mixed query environment.

Key Results

The paper presents experimental results for two state-of-the-art models, Picard and ValueNet, across three settings:

  • Setting 1 (Original Models on Spider): Models trained only on Spider showed very poor performance on temporal questions, with execution accuracy as low as 0.00% for some configurations.
  • Setting 2 (Training on TempQ4NLIDB): Training models on the TempQ4NLIDB dataset led to substantial improvements. Picard achieved up to 100% execution accuracy on temporal questions for the HR dataset.
  • Setting 3 (Blended Temporal and Non-Temporal Questions): Mixing temporal and non-temporal questions for training and testing resulted in a decline in performance compared to Setting 2, indicating the challenge of handling mixed query types. ValueNet's performance was around 39-47% in this setting.

Limitations and Future Work

  • Implicit Temporal Questions: The current dataset does not support implicit temporal questions (e.g., "last year," "5 years ago") because annotating their normalized values requires a time anchor that cannot be embedded into the data format.
  • Coverage of Temporal Question Types: The authors acknowledge that more work is needed to expand the dataset to cover other types of temporal questions.
  • Future Work: The authors plan to increase the size of their dataset and expand its coverage to include more types of temporal questions. They also aim to further investigate how models handle temporal questions.

Summary updated

BibTeX
@inproceedings{vo2022tackling,
  author = {Ngoc Phuoc An Vo and Octavian Popescu and Irene Manotas and Vadim Sheinin},
  title = {{Tackling Temporal Questions in Natural Language Interface to Databases}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
}
Figure