ECLAIR: Enhanced Clarification for Interactive Responses in an Enterprise AI Assistant

John Murzaku, Zifan Liu, Vaishnavi Muppala, Md. Mehrab Tanjim, Xiang Chen, Yunyao Li

Abstract

Large language models (LLMs) have shown remarkable progress in natural language understanding, yet they often struggle to resolve ambiguities in real-world, enterprise-level interactions. In these settings, context and domain-specific knowledge are crucial. We introduce ECLAIR (Enhanced CLArification for Interactive Responses), a multi-agent framework for interactive disambiguation. ECLAIR enhances user query clarification through a collaborative process where specialized agents detect ambiguity, reason over context, and generate targeted clarification questions. When evaluated on real-world customer data, ECLAIR demonstrates a 13% improvement in F1 score over standard few-shot methods, specifically excelling in precision and "silence logic."

Introduction

In a corporate environment, a simple query like "How many profiles are in TEST?" is fraught with ambiguity. Does "TEST" refer to a specific product segment, a dataset name, or a sandbox environment? While standard LLMs are capable of general conversation, they often stumble in these "messy" enterprise realities.

Current approaches typically rely on sequential pipelines that treat ambiguity as a purely linguistic issue, often ignoring the vital domain-specific context required for resolution. Furthermore, most research relies on synthetic datasets that fail to capture the nuances of actual industry interactions. ECLAIR addresses these gaps by providing a framework that knows when to ask for clarification and how to do so using the specific terminology of the business.

The ECLAIR Framework

ECLAIR is a modular and extendable multi-agent framework. Instead of a single prompt attempting to handle detection and generation simultaneously, ECLAIR decomposes the problem into specialized tasks:

1. Specialized Detection Agents

The framework employs four distinct agents to analyze a query from different perspectives:

  • Generic Detector: Captures domain-agnostic linguistic and lexical ambiguities.
  • Product Detector: Determines if a query refers to multiple products within a specific ecosystem (e.g., the Adobe ecosystem).
  • Entity Linking Detector: Identifies text spans that could link to multiple distinct database entities.
  • Concept Graph Module: Identifies tokens specific to the enterprise's terminology, such as terms unique to the Adobe Experience Platform (AEP).

2. Collaborative Reasoning and Generation

These agents collaboratively "vote" on whether an ambiguity exists. Their specific findings are aggregated and fed into a central LLM (GPT-3.5). This central controller decides if a clarification is truly necessary. If so, it generates a question providing the user with specific choices to resolve the uncertainty. This process includes a user feedback loop, where the user's selection is used to refine the final response.

Experimental Setup and Results

To evaluate ECLAIR, researchers utilized the AEP AI Assistant Corpus, a dataset of real-world user queries from the Adobe Experience Platform. Queries were manually annotated for contextual, syntactic, and aleatoric ambiguities. ECLAIR was compared against a GPT-3.5 baseline using ten human-crafted few-shot examples.

Performance Metrics

ECLAIR demonstrated significant improvements in precision and the ability to identify when not to interrupt the user:

  • Superior Precision: ECLAIR achieved a precision of 0.904 for "Clarification Needed," significantly outperforming the baseline's 0.732. This minimizes "over-clarification," a major friction point in user experience.
  • Enhanced Silence Logic: ECLAIR was vastly better at identifying when clarification was unnecessary, achieving a recall of 0.808 compared to the baseline's 0.214.
  • Overall Accuracy: The framework reached an average F1 score of 0.657, representing a 13% improvement over the few-shot baseline (0.520).

Key Insights

  • UX-First Design: In production, asking unnecessary questions is often worse than providing a slightly imperfect answer. ECLAIR’s high precision ensures that when the system interrupts a user, it does so for a valid, contextually grounded reason.
  • Domain Knowledge as a Filter: By integrating a Concept Graph and Entity Linking, ECLAIR identifies ambiguities that general-purpose LLMs miss. It understands that enterprise terms are pointers to complex underlying structures.
  • Modular Scalability: The agentic architecture allows developers to plug in new detectors as product lines or database schemas evolve, without requiring a full system retraining.

Limitations and Future Work

While ECLAIR excels at precision, its recall for "Clarification Needed" (0.635) was lower than the baseline (0.833). This suggests a conservative bias—the system prefers to provide a potentially incorrect answer over risking a redundant question. Future work will focus on tuning this balance and expanding the framework to handle more complex, multi-turn disambiguation scenarios across broader datasets.

Summary updated

Abstract

Large language models (LLMs) have shown remarkable progress in understanding and generating natural language across various applications. However, they often struggle with resolving ambiguities in real-world, enterprise-level interactions, where context and domain-specific knowledge play a crucial role. In this demonstration, we introduce ECLAIR (Enhanced CLArification for Interactive Responses), a multi-agent framework for interactive disambiguation. ECLAIR enhances ambiguous user query clarification through an interactive process where custom agents are defined, ambiguity reasoning is conducted by the agents, clarification questions are generated, and user feedback is leveraged to refine the final response. When tested on real-world customer data, ECLAIR demonstrates significant improvements in clarification question generation compared to standard few-shot methods.
BibTeX
@inproceedings{murzaku2025eclair,
  author = {John Murzaku and Zifan Liu and Vaishnavi Muppala and Md. Mehrab Tanjim and Xiang Chen and Yunyao Li},
  title = {{ECLAIR: Enhanced Clarification for Interactive Responses in an Enterprise AI Assistant}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=YfyqSID9G7},
}
Figure