TL;DR
ECLAIR (Enhanced CLArification for Interactive Responses) is a framework designed for enterprise AI assistants that detects ambiguity and generates clarification questions in a single, unified LLM pass. By feeding domain-grounded signals from specialized downstream agents (e.g., entity linkers, product classifiers) into the prompt, ECLAIR improves clarification precision by 17 percentage points over standard few-shot baselines, drastically reducing disruptive, unnecessary questions.
What Problem Does This Paper Address?
In conversational enterprise systems—such as the Adobe Experience Platform (AEP) AI Assistant—user queries are frequently underspecified or ambiguous (e.g., asking about a name like "TEST" without specifying whether it refers to a dataset or a segment).
Existing LLM-based disambiguation approaches suffer from two main flaws:
- Sequential, isolated pipelines: They first run a binary classifier to detect ambiguity, and only then prompt a model to ask a clarification question. This fails to leverage the direct link between why something is ambiguous and how to ask about it.
- Lack of domain context: They treat ambiguity as a purely lexical/linguistic problem, ignoring enterprise knowledge bases, schemas, and product taxonomies.
- Low precision: General models tend to over-clarify. In enterprise workflows, asking redundant clarification questions frustrates users and degrades the interaction experience.
Methods
ECLAIR introduces an extensible, multi-agent architecture paired with a single-pass prompting strategy:
User Query ──> [ Domain & Ambiguity Agents ] ──> Structured Prompt ──> Single LLM Call ──> Decision + Clarification
- Modular Downstream Agents: The query is first evaluated by specialized modules:
- Generic Ambiguity Detector: Catches domain-agnostic linguistic ambiguities.
- Product Ambiguity Detector: Identifies whether a query spans multiple Adobe products (e.g., Workfront vs. Commerce).
- Entity Linking Detector: Checks whether text spans map to multiple database entities.
- Concept Graph Module: Grounds enterprise-specific terminology and keywords into the prompt (does not detect ambiguity itself).
- Unified Prompting: Instead of a multi-step chain, all agent diagnostics and domain definitions are compiled into a structured prompt. In a single call, the LLM simultaneously makes a binary decision (Is clarification needed?) and drafts a targeted clarification question referencing specific entities or product options.
- Interactive UI Disambiguation: For production, clarification prompts are accompanied by clickable UI buttons (e.g., allowing a user to click between
Segment 'TEST'orDataset 'TEST').
Datasets
- Evaluation Corpus: 100 "hard" ambiguous user queries sampled from real-world Adobe Experience Platform (AEP) AI Assistant traffic. These represent cases that failed to resolve via standard chat-history query rewriting.
- Ambiguity Taxonomy: Queries cover three primary real-world ambiguity categories:
- Contextual: Underspecified references (e.g., "Can you show me the schema of this dataset" without defining "this").
- Syntactic: Malformed or incomplete inputs (e.g., entering raw IDs like "XYZ123").
- Aleatoric: Inherent temporal or quantitative vagueness (e.g., "Show me segments over time").
- Annotation: Each query was labeled by three annotators (with tie-breakers up to five annotators) determining whether clarification was necessary and drafting gold questions. Annotator agreement reached a 0.904 BERTScore and 0.395 ROUGE-L.
Key Insights and Ideas
- Precision matters more than recall in enterprise UX: Missing an ambiguity and attempting an answer is often less damaging than interrupting a business user with unnecessary clarification dialogues. ECLAIR intentionally optimizes for high precision.
- Keyword-heavy queries mimic search engines: Real enterprise users frequently input keywords rather than full sentences. In many syntactic cases, the system should infer intent rather than disrupt the workflow with clarification questions.
- Unification beats pipelining: Conditioning the decision and question generation simultaneously on downstream agent outputs enables the LLM to understand contextual grounding rather than guessing intent in a vacuum.
Key Results
ECLAIR was evaluated against a standard few-shot baseline (GPT-3.5 prompted with 10 human-curated examples):
| Metric | Few-Shot Baseline | ECLAIR |
|---|---|---|
| Clarification Needed — Precision | 0.732 | 0.904 (+17.2 pts) |
| Clarification Needed — Recall | 0.833 | 0.635 |
| Clarification Needed — F1 | 0.779 | 0.746 |
| Clarification NOT Needed — Precision | 0.333 | 0.438 |
| Clarification NOT Needed — Recall | 0.214 | 0.808 |
| Clarification NOT Needed — F1 | 0.261 | 0.568 |
| Macro Average F1 | 0.520 | 0.657 (+13.7 pts) |
- Question Quality: Clarification questions generated by ECLAIR aligned more closely with human reference annotations, achieving an average BERTScore of 0.886 compared to 0.868 for the few-shot baseline.
Limitations and Future Work
- Target Precision Threshold: While ECLAIR reached 90.4% precision, the team notes that enterprise production deployment requires ≥98% precision to guarantee user trust.
- Latency Overhead: Querying multiple agents prior to the LLM generation step increases response latency. Future optimizations will focus on dependency tracking and parallel agent execution.
- User Profile Awareness: Queries are often ambiguous in isolation but unambiguous when conditioned on user roles, permissions, or past preferences. The authors plan to build dedicated user-profile retrieval agents to personalize disambiguation.
Summary updated