Abstract
Open-domain conversational search (ODCS) aims to provide valuable, up-to-date information, while maintaining natural conversations to help users refine and ultimately answer information needs. However, creating an effective and robust ODCS agent is challenging. In this paper, we present a fully functional ODCS system, Ericson, which includes state-of-the-art question answering and information retrieval components, as well as intent inference and dialogue management models for proactive question refinement and recommendations. Our system was stress-tested in the Amazon Alexa Prize, by engaging in live conversations with thousands of Alexa users, thus providing empirical basis for the analysis of the ODCS system in real settings. Our interaction data analysis revealed that accurate intent classification, encouraging user engagement, and careful proactive recommendations contribute most to the users satisfaction. Our study further identifies limitations of the existing search techniques, and can serve as a building block for the next generation of ODCS agents.
Introduction
Traditional web search is often a static experience where users submit queries and receive a list of links. Open-Domain Conversational Search (ODCS) seeks to transform this into a natural, back-and-forth dialogue. The primary challenges in ODCS are managing intent uncertainty—where user requests are often vague—and maintaining conversational flow. Ericson was developed to address these issues by moving beyond simple question-answering toward proactive, multi-turn interaction.
System Architecture and Methods
Ericson utilizes a modular architecture designed for real-time response and flexibility. The core of the system is its hierarchical approach to understanding user input and managing the dialogue state.
Hierarchical Intent Classifier
A key innovation in Ericson is its two-level intent classification system. Rather than a flat model, it first employs Gradient Boosted Decision Trees (GBDT) with Word2Vec features to determine general intent. If an entity is detected, a specialized entity classifier leverages external knowledge bases, such as the Google Knowledge Graph API, to refine the context. This allows the system to distinguish between a general query and a specific request related to a known entity (e.g., identifying a name as a "singer-songwriter" to trigger music-specific modules).
Dialogue Management and Information Retrieval
The Dialogue State Manager (DSM) uses a stack-based approach to track history and decide whether to deepen the current topic or transition to a new one. The system integrates 12 specialized Information Retrieval (IR) modules, drawing from real-time sources like Wikipedia and CNN, alongside static databases for entertainment and common questions. To prevent conversational dead-ends, a Proactive Transition Component suggests "hot topics" or reminders when interaction stalls.
Evaluation and Results
The system was evaluated during the 2017 Amazon Alexa Prize, involving over 22,000 sessions and 230,000 conversational turns. This "in-the-wild" testing provided critical feedback for iterative development.
Performance Improvements
Iterative updates to the system led to a marked increase in user satisfaction. The average rating climbed from 2.4 to 3.2 on a 5-point scale over the deployment period.
Key findings include:
- Engagement Metrics: Adding "engagement techniques"—offering deeper details on a current topic—increased conversational depth by 71% for movies and 50% for news.
- Proactivity: Smart topic suggestions resulted in an 8.6% increase in user ratings and a 12.8% increase in conversation length.
- Knowledge Graph Integration: Utilizing entity-specific context was vital for correctly classifying transitions that general models often missed.
Conclusion and Future Work
Ericson demonstrates that hierarchical intent classification and proactive recommendations are essential for effective ODCS. While successful, challenges remain in making transitions feel more human-like and ensuring proactive prompts are consistently relevant. Future research will focus on enhancing dialogue naturalness and expanding the system's ability to navigate complex or sensitive topics safely.
Summary updated