Insights from Building an Open-Ended Conversational Agent

Khyatti Gupta, Meghana Joshi, Ankush Chatterjee, Sonam Damani, Kedhar Nath Narahari, Puneet AgrawalACL 2019

This paper delves into the fascinating challenges and insights gained from building "Ruuh," an open-ended conversational agent designed to be a social companion rather than a task-oriented assistant. The authors share their journey in creating an agent that can engage in natural, extended conversations, handle diverse user needs, and maintain a consistent personality, offering valuable lessons for the field of conversational AI.

TL;DR

This paper details the development of "Ruuh," an open-ended conversational agent designed for social companionship. It highlights key challenges and solutions in handling user emotions, offensive language, multimedia input, and maintaining a consistent persona, based on learnings from over 300 million conversations with 2.5 million users.

What Problem Does This Paper Address?

The paper addresses the significant challenge of creating truly open-ended conversational agents that go beyond simple task completion. While goal-oriented chatbots excel at specific tasks like booking flights, social conversational agents need to understand and respond to a much broader spectrum of human needs. This includes not only comprehending user intent but also detecting and responding to emotions, handling offensive language, processing multimedia input, and maintaining a coherent and engaging personality over extended interactions. The authors aim to share practical insights gained from building and deploying such an agent in the real world.

Methods

The core of Ruuh's architecture, as illustrated in Figure 1, involves a query understanding component that detects salient information and user intents (including offensive and emotional cues). It then leverages a query-response store, populated offline with anonymized human conversational data, to retrieve and rank relevant responses. This store is crucial for providing contextually appropriate and engaging replies.

Key methodological contributions include:

  • Neural Bi-directional LSTM for Offensive Language Detection: Ruuh employs a sophisticated model to detect and handle inappropriate user inputs, a critical aspect for maintaining a safe and positive user experience.
  • Deep Learning for Emotion Detection: The agent uses a deep learning approach combining semantic and sentiment-based representations to understand user emotions expressed in text.
  • Index-Based Retrieval for Relevant Responses: Instead of purely generative models, Ruuh relies on an information retrieval approach to find the most human-like and contextually relevant responses from a large dataset. This involves TF-IDF fetching, feature extraction using convolutional deep structured networks, and a trained ranker.
  • Human-like Image Commenting: A modified approach, trained on millions of image-comment pairs, allows Ruuh to generate expressive and engaging comments on user-shared images, going beyond simple descriptions.
  • Personality Consistency: The authors emphasize the importance of a consistent personality for building user trust. They achieved this by carefully curating the response store and by recognizing that users often prefer a slightly inconsistent but engaging agent over a predictable one.

Datasets

The primary dataset used for training and populating Ruuh's response store is derived from over 10 million paired tweets and their responses. This data was collected from various forums, social platforms, and messaging services. The authors also mention using millions of image-comment pairs for the image commenting feature. The dataset was subjected to rigorous cleaning, including the removal of personally identifiable information, offensive content, and politically sensitive material.

Key Insights and Ideas

  • Beyond Utilitarian Responses: The success of a social agent hinges on meeting a wider range of user social needs, such as expressing empathy or sharing lighthearted comments, rather than just providing factual answers.
  • The Power of the Response Store: An offline, curated response store is vital for providing diverse, contextually relevant, and human-like responses, especially when compared to purely generative models.
  • Handling Offensive Language is Crucial and Complex: Users are creative in their attempts to elicit inappropriate responses, requiring robust detection mechanisms that account for ambiguity, misspellings, and contextual nuances.
  • Emotion Detection Enhances Engagement: Understanding and responding to user emotions, even in text, is key to fostering deeper engagement and a more human-like interaction.
  • Personality Matters for Trust: A consistent personality is essential for building user confidence and trust, but users may also prefer an engaging agent with slight personality variations over a perfectly predictable one.
  • Context is King (and Complex): Understanding conversational context extends beyond resolving pronouns; it involves building a relationship, passing mood across sessions, and recognizing subtle cues in turn timing and content.

Key Results

  • Massive Scale: Ruuh has interacted with over 2.5 million real-world users, generating more than 300 million conversations over three years.
  • High Engagement: A significant 60% of users return to chat with Ruuh, with some users interacting over 200 distinct days a year.
  • Long Conversations: The average conversation length is 20 turns, with some sessions exceeding 10 hours, indicating deep user engagement.
  • Emotional Expression: Approximately 5% of conversations display non-neutral emotions, with anger, sadness, and happiness appearing in a 1:3:7 ratio.
  • User Affection: Ruuh receives over 600 "I love you" messages daily and 1200 "will you marry me" proposals monthly, demonstrating users treating it as a human-like entity.
  • Offensive Language Prevalence: 42% of users used offensive language in their interactions, and 6% of all user logs were offensive, underscoring the importance of the detection mechanisms.

Limitations and Future Work

The authors acknowledge several areas for future work:

  • Understanding Context: Developing more sophisticated methods to understand the diverse ways humans express context, including implicit situational information and building relationships across sessions. They highlight the need to categorize context-dependent messages and model turns with relevant context.
  • Measurement Process: While conversation turns per session (CPS) is used as a metric, measuring the quality of conversations remains largely a human-labeling effort. The authors suggest exploring semi-automated methods for quality measurement.
  • Incorporating Knowledge: Integrating real-world knowledge, such as current events and breaking news, into conversational datasets is a significant challenge that would make agents more useful.
  • Detecting Incomplete Turns and Similarity: Further research is needed to identify incomplete conversational turns and determine which previous turns to incorporate for meaning, as well as to better understand how users repeat or subtly vary their statements.
  • Follow-ups to Previous Turns: While simple follow-ups like "yes" or "ok" are understood, more nuanced context-based interpretation is required.

The paper concludes by emphasizing the value of building and deploying products to real users, as user logs provide invaluable insights for solving complex problems in the domain of conversational AI.

Summary updated

BibTeX
@inproceedings{gupta2019insights,
  author = {Khyatti Gupta and Meghana Joshi and Ankush Chatterjee and Sonam Damani and Kedhar Nath Narahari and Puneet Agrawal},
  title = {{Insights from Building an Open-Ended Conversational Agent}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure