TL;DR
ECLAIR (Enhanced CLArification for Interactive Responses) is a multi-agent framework designed to detect and resolve query ambiguities in enterprise AI assistants. Rather than relying solely on generic large language model (LLM) prompts, ECLAIR uses specialized downstream agents—such as entity linkers and product detectors—to decide when clarification is genuinely necessary. On real-world Adobe enterprise data, it drastically reduces unnecessary interruptions, achieving a 13% improvement in overall F1 score over standard few-shot prompting.
What Problem Does This Paper Address?
When users interact with enterprise AI assistants, their prompts are often ambiguous—for example, mentioning a term that could refer to multiple internal products, datasets, or concepts.
Existing LLM approaches typically rely on few-shot or chain-of-thought prompting in a rigid, sequential pipeline (first detect lexical ambiguity, then ask a question). These approaches treat ambiguity as purely a surface-level language issue, ignoring domain-specific knowledge and real system context. Furthermore, prior research often evaluates models on synthetic benchmarks that do not reflect messy enterprise interactions. As a result, standard assistants often "over-clarify"—interrupting the user with annoying, unnecessary clarifying questions—or fail to grasp domain terminology altogether.
Methods
ECLAIR introduces a modular multi-agent architecture integrated into the Adobe Experience Platform (AEP) AI Assistant:
- Downstream Ambiguity Agents: Instead of one monolithic prompt, incoming user queries are concurrently inspected by specialized agents:
- Sentence-Level Ambiguity Detector: Identifies domain-agnostic linguistic ambiguities.
- Product Ambiguity Detector: Determines if a query could refer to multiple Adobe AEP products.
- Entity Linking Ambiguity Detector: Spots text spans that link to multiple potential database entities (e.g., whether "TEST" refers to a segment or a dataset).
- Concept Graph Module: Flags tokens that match specialized AEP terminology.
- Aggregated Clarification Decision: If agents detect ambiguity, their findings are bundled into a prompt for a central LLM (the paper evaluates using GPT-3.5) to decide whether clarification is actually needed.
- Interactive User Resolution: When clarification is deemed necessary, ECLAIR generates a targeted question with interactive options (e.g., selectable UI chips). Once the user provides feedback, the assistant returns the refined final response.
Datasets
The authors evaluated ECLAIR on an in-house dataset sampled from real user queries in the Adobe Experience Platform (AEP) AI Assistant. The queries were annotated across three categories of ambiguity:
- Contextual: The context or object reference is underspecified.
- Syntactic: The query is malformed or incomplete, leading to indirect interpretations.
- Aleatoric: Specific tokens are undefined or hold multiple possible meanings.
Key Insights and Ideas
- Optimizing for Precision Over Recall in Enterprise UX: In real-world customer deployments, asking unnecessary clarifying questions ruins the user experience. The authors explicitly prioritize precision on the "Clarification Needed" label, ensuring the assistant only interrupts users when it has strong evidence of ambiguity.
- Domain Grounding Beats Raw LLM Reasoning: General-purpose LLMs struggle to know which entities exist in a proprietary database. Augmenting the decision pipeline with entity linking and product-classifier agents grounds the model's clarification decisions in actual system state.
- Eliminating False Alarms: Standard few-shot LLMs frequently misclassify straightforward queries as ambiguous. Introducing specialized agents provides the necessary evidence to confidently leave unambiguous queries alone.
Key Results
The authors benchmarked ECLAIR against a few-shot GPT-3.5 baseline (prompted with 10 human-crafted examples) on binary ambiguity detection (Clarification Needed vs. Clarification Not Needed):
- Overall Performance: ECLAIR improved the macro-average F1 score from 0.520 to 0.657 (a 13% absolute increase).
- Clarification Needed: ECLAIR boosted precision from 0.732 to 0.904, preventing false clarification prompts. However, recall decreased from 0.833 to 0.635 (F1 dropped slightly from 0.779 to 0.746).
- Clarification Not Needed: ECLAIR achieved major gains across the board:
- Precision rose from 0.333 to 0.438.
- Recall jumped dramatically from 0.214 to 0.808.
- F1 increased from 0.261 to 0.568.
Limitations and Future Work
- Lower Recall on Clarifications: While ECLAIR's high precision minimizes unnecessary user interruptions, its lower recall (0.635) means it occasionally fails to detect genuine ambiguities that a few-shot model would have caught.
- Preliminary Evaluation Scope: The empirical evaluation focuses strictly on the binary classification step (Clarification Needed vs. Not Needed); end-to-end multi-turn dialogue metrics and full qualitative impact on downstream task completion were not quantitatively measured.
- Future Directions: Explicit next steps are not detailed in the paper, though the authors highlight practical ongoing deployment challenges such as managing system latency and precision in live production systems.
Summary updated