TL;DR
Spoken question answering (QA) systems frequently fail when faced with the messy reality of spontaneous speech, such as disfluencies, complex syntax, and over-specificity. This paper introduces SURF (Semantic Question Reformulation), a controllable seq2seq model that rewrites failed spoken queries into cleaner, more answerable forms using linguistically-grounded operations. When evaluated on 1 million previously unanswered real-world queries, SURF successfully recovered relevant answers for up to 24% of them without requiring any expensive retraining of the underlying QA backends.
What Problem Does This Paper Address?
When people talk to voice assistants like Alexa, they use spontaneous speech filled with disfluencies ("uhm", self-corrections), colloquial phrasing, and highly specific, multi-clause structures (e.g., "my hamburger patty is in the fridge for four days should i throw it").
Because most production QA systems rely on knowledge bases derived from formal written text, these messy spoken queries often cause retrieval failures, leaving users with an unsatisfying "I don't know" response. Retraining or fine-tuning massive, federated QA backends to handle every quirk of spoken language is prohibitively expensive and computationally impractical. The authors tackle this challenge by introducing a lightweight, modular question-rewriting preprocessing layer that bridges the gap between conversational speech and formal knowledge retrieval.
Methods
The authors propose SURF, a single seq2seq Transformer model (based on BART-base) trained to perform controllable text generation via prefix tokens. By prepending a specific token to the input query, SURF can execute three distinct, linguistically-informed reformulation operations:
- Question Repair (
REP): Removes disfluencies, fixes grammatical errors, and increases formality via lexical substitution (e.g., rewriting "Where can I get a booze after 11 pm?" to "Which stores sell beer after 11 pm?"). - Root Wh- Transform (
ROO): Restructures complex or polar (yes/no) questions so that the interrogative wh- phrase sits at the root of the sentence, making it easier for standard QA systems to parse (e.g., converting "Is Sherlock Holmes a real person?" to "Who is Sherlock Holmes?"). - Question Generalization (
GEN): Relaxes semantic constraints by dropping adjectives or replacing highly specific entities with hypernyms or holonyms to broaden search recall (e.g., generalizing "Do poisonous pythons live in Miami?" to "Do snakes live in Florida?").
SURF is trained in a two-stage pipeline: first, a large-scale weakly-supervised pre-training phase helps the model learn basic repairs and syntactic restructuring. Second, a high-quality manual fine-tuning phase teaches the model complex operations like generalization.
Datasets
- Pre-training Data: A weakly-supervised dataset of 1.2 million samples derived from the public Multi-domain Question Rewriting (MQR) corpus. The authors used a rule-based question-type classifier to automatically assign
REPandROOtask labels. - Fine-tuning Data: A newly curated dataset of 3,851 questions with human-annotated reformulations spanning all three target operations.
- Evaluation Data: A massive, representative test set consisting of 1 million real-world unanswered questions sampled from the live traffic of a leading commercial voice assistant.
Key Insights and Ideas
- Syntactic Reshaping Beats Simple Repair: While fixing typos and removing disfluencies (
REP) provides a steady baseline improvement, transforming the core question structure (ROO)—especially turning hard-to-answer polar "yes/no" questions into factoid "wh-" questions—unlocks significantly higher retrieval success. - Generalization as a Safety Net: When a query is too specific to find an exact match, relaxing constraints to return a highly related "superset" answer is vastly preferred by users over a dead-end failure.
- Complementary Operations: The different reformulation strategies are highly orthogonal. A query that fails under simple repair might succeed beautifully when generalized or reshaped, meaning a combined or adaptive strategy yields the best overall performance.
Key Results
- Substantial Answer Rate Recovery: Because the evaluation dataset consisted entirely of previously unanswered questions, the baseline answer rate was 0.00%. SURF-
REPachieved a 9.41% answer rate, SURF-ROOreached 13.18%, and the combinedROO+GENpipeline hit 17.92%. - The Upper Bound: If the system could optimally choose the single best reformulation per question (
OPTIMAL), the answer rate jumped to 24.15%, effectively recovering a quarter of all dead queries. - High Relevance: Human evaluation confirmed that the answers retrieved weren't just random shots in the dark; 73% of the answers retrieved via the
ROO+GENreformulations were either exact matches or highly related to the user's original intent. - Production Deployment: The SURF-
ROOconfiguration was successfully deployed to a live commercial voice assistant, successfully resolving millions of previously unanswerable customer requests daily.
The breakdown of how different rewrite operators perform across various question types can be seen below:
Limitations and Future Work
- Pipeline Latency: Applying multiple operations sequentially (such as running
ROOand then passing the output toGEN) increases inference latency and introduces the risk of cascading errors. The authors note that future iterations will focus on training the model to perform joint multi-operation rewrites in a single pass. - LLM Constraints: The authors deliberately avoided massive Large Language Models (LLMs) due to strict production latency requirements and data confidentiality restrictions associated with commercial APIs.
- Language Scope: The current implementation and evaluation framework are strictly limited to English-language queries, leaving cross-lingual transfer as an open area for research.
Summary updated