TL;DR
This paper introduces an interactive semantic parsing system for OpenStreetMap (OSM) queries that explicitly detects parsing errors and generates targeted clarification questions for human users. By combining uncertainty detection with multi-source training, the system significantly improves parsing accuracy, making complex geospatial data more accessible.
What Problem Does This Paper Address?
Semantic parsing, the task of converting natural language queries into formal database queries, faces unique challenges when dealing with large, complex, and often ambiguous real-world databases like OpenStreetMap (OSM). Unlike structured databases, OSM data can have fuzzy definitions, open-ended lists, and inconsistent tagging, meaning there isn't always a single "correct" answer. This ambiguity, coupled with the difficulty of creating comprehensive gold-standard datasets, necessitates an interactive approach. Previous interactive systems often relied on users guessing where the parser went wrong. This paper addresses the need for a more direct and informative interaction, where the parser actively identifies potential errors or ambiguities and asks specific questions to resolve them.
Methods
The core of this work is an "error-aware" interactive semantic parsing system. The approach involves:
- Error Detection: The system uses entropy-based uncertainty measures to pinpoint potential ambiguities or errors in its generated parse. This is done at a token level, analyzing the probability distribution of possible tokens.
- Clarification Question Generation: Based on the detected uncertainty, the system automatically generates concise clarification questions. These questions are designed to directly address the suspected source of ambiguity. For instance, if the parser is uncertain between "wine" and "alcohol" for an "Off License" query, it will ask "Did you mean wine or alcohol?".
- Multi-Source Training: The parser is trained using a multi-source neural network architecture. This allows it to incorporate not only the original user query but also the generated clarification question and the human user's answer as additional inputs. This "dialogue" information helps the parser refine its understanding and improve future parses.
- Beam Search: Beam search is employed to generate alternative parses and potential clarification question answers, providing a richer set of options for the user to choose from or to inform the model's learning.
Datasets
- NLmaps v2 dataset: This dataset, built on the Overpass API for OSM, forms the basis of the work. It includes template-based expansions, which the authors note introduced some problematic features and duplicates.
- Dialogue-Enriched Data: The authors created a dataset of 15,000 clarification questions, answered by extracting information from gold standard parses. They also collected a smaller dataset of 960 examples where human users provided answers to automatically generated questions.
Key Insights and Ideas
- Proactive Error Identification: The most significant insight is the shift from passive user feedback to active, parser-driven error detection and clarification. This makes the interaction more efficient and less guesswork for the user.
- Targeted Clarification is Key: Generating specific questions that pinpoint the ambiguity (e.g., "wine or alcohol?") is far more effective than generic prompts.
- Multi-Source Learning for Dialogue: Incorporating the dialogue history (clarification questions and user answers) as input to the neural network significantly boosts performance, demonstrating the value of contextual interaction.
- Entropy as a Reliable Uncertainty Signal: Entropy-based measures effectively identify points of ambiguity in the parsing process, serving as a strong signal for generating helpful clarification questions.
Key Results
- The proposed error-aware interactive system achieved an improvement of 1.2% F1 score over a strong baseline parser that already performed at 90.26% on the NLmaps dataset.
- A multi-source model incorporating parse hypothesis, clarification question, and logged user answer as additional inputs showed statistically significant improvements in accuracy and F1 score (e.g., over 15% gain on human-annotated data when the baseline parser failed).
- The system's ability to generate concise and informative clarification questions was validated through a human interaction study.
Limitations and Future Work
- Dataset Filtering: The authors note that the human annotation dataset was filtered to include only examples where the baseline parser failed. This might inflate performance gains on this specific subset.
- Offline Learning: The current system learns from user feedback in an offline manner.
- Future Work: The authors suggest moving towards online learning in interactive semantic parsing, where the model can adapt and improve in real-time during the interaction. They also highlight the potential for further improvements in precise communication and negotiation of answers.
Summary updated