Detecting Ambiguities to Guide Query Rewrite for Robust Conversations in Enterprise AI Assistants

Md. Mehrab Tanjim, Xiang Chen, Victor S. Bursztyn, Uttaran Bhattacharya, Tung Mai, Vaishnavi Muppala, Akash Maharaj, Saayan Mitra, Eunyee Koh, Yunyao Li, Ken RussellarXiv 2025 · DOI · arXiv

Abstract

Enterprise AI assistants often struggle with underspecified user queries in multi-turn conversations. While Large Language Models (LLMs) can rewrite these queries to restore context, "always-on" rewriting often leads to hallucinations and errors in clear queries. This paper proposes an Ambiguity-guided Query Rewrite framework. By employing a lightweight Small Language Model (SLM) to detect ambiguity before triggering a rewrite, the system improves accuracy, reduces downstream errors from 18% to 8%, and significantly lowers computational costs.

Introduction

In multi-turn enterprise dialogues, users frequently rely on context from previous turns. For instance, after asking "What is a segment?", a user might follow up with "How many do I have?". Without context, the second query is ambiguous. The standard industry solution is Query Rewrite (QR), where an LLM reformulates the query into a standalone question (e.g., "How many segments do I have?").

However, the authors identify a "hallucination trap": when an LLM rewrites an already clear query containing specific identifiers (e.g., "What is the ID of the 'ABC' dataset?"), it may inadvertently alter technical names, causing system failures. Moreover, universal LLM rewriting is computationally expensive and adds latency.

Comparison of different query handling strategies: Always Rewriting, No Rewrite, and the proposed Ambiguity-guided Query Rewrite.

Proposed Methodology: The NLU-NLG Framework

The researchers introduce a two-stage framework that balances efficiency and robustness:

  1. Ambiguity Detection (NLU Component): Instead of a heavy LLM, a "Small Language Model" (SLM) classifier is used. This classifier integrates text embeddings from a Sentence Transformer with hand-picked linguistic features such as query length, referential counts (tracking pronouns like "it" or "this"), and the Coleman-Liau readability index. It also utilizes regex-based lexical rules to identify missing or present business entities.
  2. Query Rewrite (NLG Component): If the SLM flags a query as ambiguous, it is passed to an LLM (e.g., GPT-3.5 or Llama-3) for contextual rewriting based on the last five conversation turns. Clear queries bypass this stage entirely, preserving their original precision.
The proposed pipeline architecture and the internal structure of the Ambiguity Detecting Small Language Model.

Experimental Setup and Data

The framework was evaluated using a combination of real-world and synthetic data:

  • Real-world Logs: 3,000 anonymized logs from the Adobe Experience Platform (AEP) AI Assistant, expert-annotated for ambiguity.
  • Synthetic Augmentation: 1,372 synthetic examples created by programmatically introducing ambiguity into clear queries (e.g., replacing nouns with pronouns).
  • Golden Rewrites: 366 multi-turn conversations with human-verified "perfect" rewrites for benchmarking.

Key Insights and Results

The study revealed that ambiguities are primarily Pragmatic (context-dependent, 63.5%), followed by Syntactic (31.9%) and Lexical (4.5%). This distribution underscores the importance of tracking contextual cues like referential pronouns.

Key performance metrics include:

  • Superior Detection: The hybrid SLM classifier achieved an F1-score of 90.19, significantly outperforming few-shot LLM baselines (e.g., GPT-3.5 at 71.58 F1), which tended to over-classify queries as ambiguous.
  • Improved Downstream Accuracy: The ambiguity-guided approach surpassed the "Always Rewrite" baseline across semantic metrics (BLEU, BERTScore).
  • Operational Efficiency: The classifier's latency is a mere 0.01 seconds, making it highly suitable for production environments where LLM latency is a bottleneck.
  • Real-World Impact: Deployment in the AEP AI Assistant reduced downstream task errors (such as SQL generation) from 18% to 8%.

Conclusion

The "Ambiguity-guided Query Rewrite" framework demonstrates that selective intervention is more robust than universal rewriting. By bypassing the rewrite module for clear queries, the system avoids the "hallucination trap" and preserves technical identifiers. Future work will focus on adapting lexical rules to new domains and addressing complex pragmatic ambiguities where user intent shifts mid-conversation.

Summary updated

Abstract

Multi-turn conversations with an Enterprise AI Assistant can be challenging due to conversational dependencies in questions, leading to ambiguities and errors. To address this, we propose an NLU-NLG framework for ambiguity detection and resolution through reformulating query automatically and introduce a new task called "Ambiguity-guided Query Rewrite." To detect ambiguities, we develop a taxonomy based on real user conversational logs and draw insights from it to design rules and extract features for a classifier which yields superior performance in detecting ambiguous queries, outperforming LLM-based baselines. Furthermore, coupling the query rewrite module with our ambiguity detecting classifier shows that this end-to-end framework can effectively mitigate ambiguities without risking unnecessary insertions of unwanted phrases for clear queries, leading to an improvement in the overall performance of the AI Assistant. Due to its significance, this has been deployed in the real world application, namely Adobe Experience Platform AI Assistant.
BibTeX
@misc{tanjim2025detecting,
  author = {Md. Mehrab Tanjim and Xiang Chen and Victor S. Bursztyn and Uttaran Bhattacharya and Tung Mai and Vaishnavi Muppala and Akash Maharaj and Saayan Mitra and Eunyee Koh and Yunyao Li and Ken Russell},
  title = {{Detecting Ambiguities to Guide Query Rewrite for Robust Conversations in Enterprise AI Assistants}},
  year = {2025},
  eprint = {2502.00537},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.00537},
}
Figure