Abstract
Understanding search queries is a hard problem as it involves dealing with "word salad" text ubiquitously issued by users. However, if a query resembles a well-formed question, a natural language processing pipeline is able to perform more accurate interpretation, thus reducing downstream compounding errors. Hence, identifying whether or not a query is well formed can enhance query understanding. Here, we introduce a new task of identifying a well-formed natural language question. We construct and release a dataset of 25,100 publicly available questions classified into well-formed and non-wellformed categories and report an accuracy of 70.7% on the test set. We also show that our classifier can be used to improve the performance of neural sequence-to-sequence models for generating questions for reading comprehension.
Introduction
Most natural language processing (NLP) tools are optimized for formal, "clean" text like news articles or books. In contrast, real-world search engine queries often manifest as "word salad"—fragments like "headache evenings?" or "population of owls just in north america?" These lack the grammatical structure required for traditional NLP pipelines to work effectively.
While some research focuses on paraphrasing these messy queries into proper questions, there hasn't been a standard way to determine if a query is already a well-formed question. Identifying well-formed questions (e.g., "What causes headaches during evenings?") allows systems to route them through more sophisticated linguistic tools, leading to better answers in search engines, chatbots, and personal assistants.
Methods
The authors treat well-formedness identification as a binary classification problem. They developed a feed-forward neural network with two hidden layers to predict the probability $p_{wf}(q)$ that a query $q$ is well-formed.
The model utilizes a combination of lexical and syntactic features:
- Lexical Features: Word unigrams and bigrams, along with character 3-grams and 4-grams to mitigate the impact of spelling errors.
- Syntactic Features: Part-of-Speech (POS) tags (unigrams, bigrams, and trigrams) to capture underlying grammatical structure.
Each feature is represented as a real-valued embedding. These embeddings are summed by type and then concatenated to form the input layer of the network.
The Well-formedness Dataset
The primary source of queries is the Paralex Corpus, which contains noisy paraphrase pairs from WikiAnswers. From this, the authors created a new dataset of 25,100 queries. Each query was labeled by five crowdworkers based on three criteria: grammaticality, whether it is an explicit question, and the absence of spelling errors.
The dataset includes a "well-formedness probability" based on the average of human labels. Human agreement on what constitutes a "well-formed" question is relatively high, with a Fleiss’ kappa of 0.52.
Key Results
The classification experiments demonstrate that syntax is a critical component of well-formedness. Adding POS-tag features provided a significant 5.2 point boost in accuracy over lexical features alone, confirming that well-formedness is deeply tied to grammatical structure.
The best-performing model achieved 70.7% accuracy on the test set, outperforming the majority class baseline of 61.5%. However, a significant gap remains compared to the approximate human upper bound of 88.4%, suggesting room for improvement in neural linguistic intuition.
Beyond classification, the authors demonstrated the utility of this signal in a downstream question generation task using the SQuAD dataset. By using the classifier as a reranker for a sequence-to-sequence model, they improved the BLEU-4 score from 12.0 to 12.2. Qualitative analysis showed the reranker successfully prioritized natural-sounding questions over awkward or truncated candidates.
Discussion and Future Work
Despite its successes, the model faces limitations in deep semantics and long-term dependencies. For instance, it may misclassify "What is the history of dirk bikes?" as well-formed because "dirk" is a valid noun, missing the semantic error where "dirt" was intended. It also occasionally fails to catch tense mismatches across distant words.
Future work could leverage Transformer-based architectures to better capture long-range dependencies and semantic nuances. Additionally, improving sensitivity to spelling errors that alter a word's Part-of-Speech remains a key challenge for future iterations.
Summary updated