Abstract
Enterprise AI assistants often struggle with underspecified user queries in multi-turn conversations. While Large Language Models (LLMs) can rewrite these queries to restore context, "always-on" rewriting often leads to hallucinations and errors in clear queries. This paper proposes an Ambiguity-guided Query Rewrite framework. By employing a lightweight Small Language Model (SLM) to detect ambiguity before triggering a rewrite, the system improves accuracy, reduces downstream errors from 18% to 8%, and significantly lowers computational costs.
Introduction
In multi-turn enterprise dialogues, users frequently rely on context from previous turns. For instance, after asking "What is a segment?", a user might follow up with "How many do I have?". Without context, the second query is ambiguous. The standard industry solution is Query Rewrite (QR), where an LLM reformulates the query into a standalone question (e.g., "How many segments do I have?").
However, the authors identify a "hallucination trap": when an LLM rewrites an already clear query containing specific identifiers (e.g., "What is the ID of the 'ABC' dataset?"), it may inadvertently alter technical names, causing system failures. Moreover, universal LLM rewriting is computationally expensive and adds latency.
Proposed Methodology: The NLU-NLG Framework
The researchers introduce a two-stage framework that balances efficiency and robustness:
- Ambiguity Detection (NLU Component): Instead of a heavy LLM, a "Small Language Model" (SLM) classifier is used. This classifier integrates text embeddings from a Sentence Transformer with hand-picked linguistic features such as query length, referential counts (tracking pronouns like "it" or "this"), and the Coleman-Liau readability index. It also utilizes regex-based lexical rules to identify missing or present business entities.
- Query Rewrite (NLG Component): If the SLM flags a query as ambiguous, it is passed to an LLM (e.g., GPT-3.5 or Llama-3) for contextual rewriting based on the last five conversation turns. Clear queries bypass this stage entirely, preserving their original precision.
Experimental Setup and Data
The framework was evaluated using a combination of real-world and synthetic data:
- Real-world Logs: 3,000 anonymized logs from the Adobe Experience Platform (AEP) AI Assistant, expert-annotated for ambiguity.
- Synthetic Augmentation: 1,372 synthetic examples created by programmatically introducing ambiguity into clear queries (e.g., replacing nouns with pronouns).
- Golden Rewrites: 366 multi-turn conversations with human-verified "perfect" rewrites for benchmarking.
Key Insights and Results
The study revealed that ambiguities are primarily Pragmatic (context-dependent, 63.5%), followed by Syntactic (31.9%) and Lexical (4.5%). This distribution underscores the importance of tracking contextual cues like referential pronouns.
Key performance metrics include:
- Superior Detection: The hybrid SLM classifier achieved an F1-score of 90.19, significantly outperforming few-shot LLM baselines (e.g., GPT-3.5 at 71.58 F1), which tended to over-classify queries as ambiguous.
- Improved Downstream Accuracy: The ambiguity-guided approach surpassed the "Always Rewrite" baseline across semantic metrics (BLEU, BERTScore).
- Operational Efficiency: The classifier's latency is a mere 0.01 seconds, making it highly suitable for production environments where LLM latency is a bottleneck.
- Real-World Impact: Deployment in the AEP AI Assistant reduced downstream task errors (such as SQL generation) from 18% to 8%.
Conclusion
The "Ambiguity-guided Query Rewrite" framework demonstrates that selective intervention is more robust than universal rewriting. By bypassing the rewrite module for clear queries, the system avoids the "hallucination trap" and preserves technical identifiers. Future work will focus on adapting lexical rules to new domains and addressing complex pragmatic ambiguities where user intent shifts mid-conversation.
Summary updated