Abstract
We present ECLAIR (Enhanced CLArification for Interactive Responses), a novel unified and end-to-end framework for interactive disambiguation in enterprise AI assistants. ECLAIR generates clarification questions for ambiguous user queries and resolves ambiguity based on the user's response. We introduce a generalized architecture capable of integrating ambiguity information from multiple downstream agents, enhancing context-awareness in resolving ambiguities and allowing enterprise-specific definition of agents. We further define agents within our system that provide domain-specific grounding information. We conduct experiments comparing ECLAIR to few-shot prompting techniques and demonstrate ECLAIR's superior performance in clarification question generation and ambiguity resolution.
Introduction
In the context of enterprise AI assistants, user queries are frequently underspecified or ambiguous. A request such as "Show me segments over time" lacks the necessary specificity regarding which segments or what time period the user intends to analyze. While humans intuitively seek clarification, standard Large Language Models (LLMs) often struggle to identify these ambiguities or provide relevant follow-up questions without specific domain context.
ECLAIR (Enhanced CLArification for Interactive Responses) addresses these challenges by providing a unified, end-to-end framework designed to detect and resolve ambiguity. By integrating domain-specific knowledge through a modular "agentic" architecture and employing a single-pass prompting strategy, ECLAIR significantly outperforms standard LLM approaches in both ambiguity detection accuracy and the quality of generated clarification questions.
The Problem of Ambiguity in Enterprise AI
Current systems for handling query ambiguity typically suffer from three primary limitations:
- Sequential Inefficiency: Many systems utilize a pipeline approach where ambiguity detection and question generation are separate steps. This fails to capture the interdependence between the nature of the ambiguity and the optimal way to phrase a clarification.
- Context Blindness: General-purpose LLMs lack the "tribal knowledge" inherent to specific organizations. They may not recognize that a term like "ABC" refers to a proprietary dataset or a specific marketing segment within an enterprise platform.
- Synthetic Data Bias: Much of the existing research relies on synthetic datasets that do not reflect the complexity and unpredictability of real-world business queries.
The ECLAIR Framework
ECLAIR introduces a modular architecture that leverages specialized downstream agents to provide grounding for the LLM. Instead of requiring a single model to possess all domain knowledge, ECLAIR queries several agents:
- Entity Linking Agent: Determines if query terms map to multiple database entries.
- Product Ambiguity Detector: Identifies if a query refers to multiple different software products.
- Concept Graph Module: Provides grounding for enterprise-specific terminology.
- Sentence-Level Detector: Identifies general linguistic or structural ambiguities.
The Unified Prompt Innovation
Unlike traditional methods that require multiple LLM calls, ECLAIR aggregates the outputs from these agents into a single, comprehensive prompt. The LLM then performs a binary decision (determining if the query is ambiguous) and generates the clarification question in a single pass. This "one-shot" approach allows the model to synthesize the relationship between the detected ambiguity and the necessary resolution more effectively.
Experimental Evaluation
The researchers evaluated ECLAIR using real-world user data consisting of 100 "hard case" queries from the Adobe Experience Platform (AEP) AI Assistant. These queries were anonymized and manually annotated by human experts to create a "gold standard" for evaluation. While the framework was also compared against standard benchmarks like ClariQ and Ambiguous TriviaQA, the primary focus remained on the complex, enterprise-specific contexts that these public datasets often lack.
Key Results and Insights
ECLAIR demonstrated significant performance gains over standard few-shot prompting baselines:
- Precision Boost: ECLAIR improved the precision of "Clarification Needed" decisions by 17 points (0.904 vs. 0.732). In a production environment, high precision is critical to avoid "over-clarifying," which can frustrate users.
- Overall Performance: The framework achieved an overall F1 score of 0.657, representing a 13% improvement over the baseline.
- Human Alignment: Using BERTScore to measure semantic similarity, the researchers found that ECLAIR’s generated questions aligned much more closely with human-written gold standards than the baseline.
- Domain Grounding: The results highlight that domain grounding via agents is a viable alternative to massive fine-tuning. Providing the LLM with "hints" from specialized agents allows for professional and contextually accurate responses.
Conclusion and Future Work
ECLAIR represents a significant step forward in making enterprise AI assistants more interactive and reliable. By treating ambiguity detection and question generation as interdependent tasks grounded in domain knowledge, the framework provides a more seamless user experience.
Future work will focus on reducing the latency associated with executing multiple agents and complex LLM calls through parallel execution. Additionally, the team aims to incorporate user personalization—resolving ambiguity based on an individual's specific work history—and exploring optimal clarification strategies, such as providing a "best-guess" answer alongside a clarification option.
Summary updated