Identifying Well-formed Natural Language Questions

Manaal Faruqui, Dipanjan Das

Introduction

Most Natural Language Processing (NLP) systems are optimized for "clean" text found in structured sources like news articles or Wikipedia. However, real-world search engine queries often deviate significantly from these standards, frequently appearing as "word salads" or fragmented phrases. For example, a user might type "headache evenings?" instead of asking "What causes headaches in the evening?". These ill-formed queries create "compounding errors" in downstream NLP pipelines, making it difficult for automated systems to accurately extract user intent.

To address this, researchers at Google AI Language have introduced a new task: automatically identifying whether a search query is a "well-formed" natural language question. By distinguishing "NLP-ready" questions from messy fragments, developers can build more robust chatbots, personal assistants, and search recommendation engines.

Defining Well-formedness

The researchers define a well-formed question based on three primary criteria: grammaticality, whether it is an explicit question, and the absence of spelling errors.

Examples of well-formed and ill-formed queries with reasoning.

To facilitate this task, a new benchmark dataset was created from the Paralex corpus, containing 25,100 unique queries. Each query was annotated by five crowdworkers. The final well-formedness probability ($p_{wf}$) for a query is the average of these five ratings. The dataset has been publicly released to encourage further research in query understanding.

Methodology

The authors developed a classification model using a feed-forward neural network with two hidden layers and ReLU activations. Rather than relying on raw text alone, the model utilizes a multi-faceted feature engineering approach:

  1. Lexical Features: Word unigrams and bigrams.
  2. Character-level Features: 3- and 4-grams to help identify spelling mistakes.
  3. Syntactic Features: Part-of-Speech (POS) n-grams (1, 2, and 3-grams) generated via Google’s SyntaxNet.

Key Results and Insights

The experimental results demonstrate that structural information is a critical indicator of well-formedness. The inclusion of POS n-grams provided a significant accuracy boost over models using words alone, suggesting that the grammatical structure of a query is often more telling than its specific topic.

Classification accuracy across different model configurations.

The best-performing model, combining word and POS features, achieved an accuracy of 70.7%. This significantly outperformed the "question word" baseline (54.9%), which simply checks if a query starts with words like "Who" or "What." However, a gap remains between machine performance and the "human upper bound" of 88.4%, indicating that current models still struggle with complex linguistic reasoning.

Beyond simple classification, the researchers demonstrated the practical utility of this model by using it as a reranker for a sequence-to-sequence (seq2seq) question generation task. By selecting candidates with higher well-formedness scores, the system improved its BLEU-4 score from 12.0 to 12.2, effectively filtering out ungrammatical "hallucinations" common in generative models.

Limitations and Future Directions

The current model faces challenges in two main areas: deep semantics and long-term dependencies. It struggles to identify contextually incorrect words that are otherwise valid nouns (e.g., "dirk bikes" vs. "dirt bikes") and often misses tense inconsistencies across longer sentences (e.g., "What sort of work did Edvard Munch made?"). Future research may leverage transformer-based architectures to better capture these complex syntactic and semantic relationships.

Summary updated

Abstract

Understanding search queries is a hard problem as it involves dealing with "word salad" text ubiquitously issued by users. However, if a query resembles a well-formed question, a natural language processing pipeline is able to perform more accurate interpretation, thus reducing downstream compounding errors. Hence, identifying whether or not a query is well formed can enhance query understanding. Here, we introduce a new task of identifying a well-formed natural language question. We construct and release a dataset of 25,100 publicly available questions classified into well-formed and non-wellformed categories and report an accuracy of 70.7% on the test set. We also show that our classifier can be used to improve the performance of neural sequence-to-sequence models for generating questions for reading comprehension.
BibTeX
@inproceedings{faruqui2018identifying,
  author = {Manaal Faruqui and Dipanjan Das},
  title = {{Identifying Well-formed Natural Language Questions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1091},
}
Figure