Answering Unanswered Questions through Semantic Reformulations in Spoken QA

Pedro Faustini, Zhiyu Chen, Besnik Fetahu, Oleg Rokhlenko, Shervin Malmasi

TL;DR

Spoken question answering (QA) systems frequently fail when faced with the messy reality of spontaneous speech, such as disfluencies, complex syntax, and over-specificity. This paper introduces SURF (Semantic Question Reformulation), a controllable seq2seq model that rewrites failed spoken queries into cleaner, more answerable forms using linguistically-grounded operations. When evaluated on 1 million previously unanswered real-world queries, SURF successfully recovered relevant answers for up to 24% of them without requiring any expensive retraining of the underlying QA backends.

What Problem Does This Paper Address?

When people talk to voice assistants like Alexa, they use spontaneous speech filled with disfluencies ("uhm", self-corrections), colloquial phrasing, and highly specific, multi-clause structures (e.g., "my hamburger patty is in the fridge for four days should i throw it").

Because most production QA systems rely on knowledge bases derived from formal written text, these messy spoken queries often cause retrieval failures, leaving users with an unsatisfying "I don't know" response. Retraining or fine-tuning massive, federated QA backends to handle every quirk of spoken language is prohibitively expensive and computationally impractical. The authors tackle this challenge by introducing a lightweight, modular question-rewriting preprocessing layer that bridges the gap between conversational speech and formal knowledge retrieval.

Methods

The authors propose SURF, a single seq2seq Transformer model (based on BART-base) trained to perform controllable text generation via prefix tokens. By prepending a specific token to the input query, SURF can execute three distinct, linguistically-informed reformulation operations:

  1. Question Repair (REP): Removes disfluencies, fixes grammatical errors, and increases formality via lexical substitution (e.g., rewriting "Where can I get a booze after 11 pm?" to "Which stores sell beer after 11 pm?").
  2. Root Wh- Transform (ROO): Restructures complex or polar (yes/no) questions so that the interrogative wh- phrase sits at the root of the sentence, making it easier for standard QA systems to parse (e.g., converting "Is Sherlock Holmes a real person?" to "Who is Sherlock Holmes?").
  3. Question Generalization (GEN): Relaxes semantic constraints by dropping adjectives or replacing highly specific entities with hypernyms or holonyms to broaden search recall (e.g., generalizing "Do poisonous pythons live in Miami?" to "Do snakes live in Florida?").

SURF is trained in a two-stage pipeline: first, a large-scale weakly-supervised pre-training phase helps the model learn basic repairs and syntactic restructuring. Second, a high-quality manual fine-tuning phase teaches the model complex operations like generalization.

Datasets

  • Pre-training Data: A weakly-supervised dataset of 1.2 million samples derived from the public Multi-domain Question Rewriting (MQR) corpus. The authors used a rule-based question-type classifier to automatically assign REP and ROO task labels.
  • Fine-tuning Data: A newly curated dataset of 3,851 questions with human-annotated reformulations spanning all three target operations.
  • Evaluation Data: A massive, representative test set consisting of 1 million real-world unanswered questions sampled from the live traffic of a leading commercial voice assistant.

Key Insights and Ideas

  • Syntactic Reshaping Beats Simple Repair: While fixing typos and removing disfluencies (REP) provides a steady baseline improvement, transforming the core question structure (ROO)—especially turning hard-to-answer polar "yes/no" questions into factoid "wh-" questions—unlocks significantly higher retrieval success.
  • Generalization as a Safety Net: When a query is too specific to find an exact match, relaxing constraints to return a highly related "superset" answer is vastly preferred by users over a dead-end failure.
  • Complementary Operations: The different reformulation strategies are highly orthogonal. A query that fails under simple repair might succeed beautifully when generalized or reshaped, meaning a combined or adaptive strategy yields the best overall performance.

Key Results

  • Substantial Answer Rate Recovery: Because the evaluation dataset consisted entirely of previously unanswered questions, the baseline answer rate was 0.00%. SURF-REP achieved a 9.41% answer rate, SURF-ROO reached 13.18%, and the combined ROO+GEN pipeline hit 17.92%.
  • The Upper Bound: If the system could optimally choose the single best reformulation per question (OPTIMAL), the answer rate jumped to 24.15%, effectively recovering a quarter of all dead queries.
  • High Relevance: Human evaluation confirmed that the answers retrieved weren't just random shots in the dark; 73% of the answers retrieved via the ROO+GEN reformulations were either exact matches or highly related to the user's original intent.
  • Production Deployment: The SURF-ROO configuration was successfully deployed to a live commercial voice assistant, successfully resolving millions of previously unanswerable customer requests daily.

The breakdown of how different rewrite operators perform across various question types can be seen below:

Figure 2: Answer rate of different reformulation tasks grouped by original question types.

Limitations and Future Work

  • Pipeline Latency: Applying multiple operations sequentially (such as running ROO and then passing the output to GEN) increases inference latency and introduces the risk of cascading errors. The authors note that future iterations will focus on training the model to perform joint multi-operation rewrites in a single pass.
  • LLM Constraints: The authors deliberately avoided massive Large Language Models (LLMs) due to strict production latency requirements and data confidentiality restrictions associated with commercial APIs.
  • Language Scope: The current implementation and evaluation framework are strictly limited to English-language queries, leaving cross-lingual transfer as an open area for research.

Summary updated

Abstract

Spoken Question Answering (QA) is a key feature of voice assistants, usually backed by multiple QA systems. Users ask questions via spontaneous speech which can contain disfluencies, errors, and informal syntax or phrasing. This is a major challenge in QA, causing unanswered questions or irrelevant answers, and leading to bad user experiences. We analyze failed QA requests to identify core challenges: lexical gaps, proposition types, complex syntactic structure, and high specificity. We propose a Semantic Question Reformulation (SURF) model offering three linguistically-grounded operations (repair, syntactic reshaping, generalization) to rewrite questions to facilitate answering. Offline evaluation on 1M unanswered questions from a leading voice assistant shows that SURF significantly improves answer rates: up to 24% of previously unanswered questions obtain relevant answers (75%). Live deployment shows positive impact for millions of customers with unanswered questions; explicit relevance feedback shows high user satisfaction.
BibTeX
@inproceedings{faustini2023answering,
  author = {Pedro Faustini and Zhiyu Chen and Besnik Fetahu and Oleg Rokhlenko and Shervin Malmasi},
  title = {{Answering Unanswered Questions through Semantic Reformulations in Spoken QA}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-industry.70},
}
Figure