Automated Support for Collective Memory of Conversational Interactions

Walter Stephen Lasecki, Jeffrey Philip Bigham

Abstract

Crowd-powered conversational assistants leverage groups of human workers to provide nuanced interactions. However, these systems often suffer from "collective amnesia" because the rotating cast of workers lacks access to a persistent memory of past user interactions. This paper presents Chorus:Mnemonic, a system designed to provide long-term memory for crowd-powered agents. By combining automated topic modeling with human-in-the-loop fact curation, Chorus:Mnemonic surfaces relevant historical context to workers in real-time. This approach maintains conversational consistency, reduces cognitive load, and enhances user privacy by siloing information based on current relevance.

Introduction

Crowd-powered systems like Chorus utilize a pool of transient human workers to act as a single conversational entity. While effective for complex tasks, this model leads to a lack of continuity; a worker participating in a session today is unlikely to have been present during previous interactions. Consequently, the system frequently "forgets" user preferences, medical constraints (e.g., allergies), or the status of ongoing tasks.

Existing methods for memory management, such as shared notepads, fail to scale. As conversations lengthen, the volume of recorded facts imposes a high cognitive load on workers who must quickly digest history to respond. Furthermore, exposing a user's entire history to every temporary worker poses significant privacy risks. There is a critical need for a system that can selectively retrieve and present only the most pertinent information.

System Design: Chorus:Mnemonic

Chorus:Mnemonic addresses the challenge of collective memory through a multi-stage pipeline that integrates automated retrieval with human oversight.

Fact Curation and Management

The system separates the act of conversation from the management of memory. Specific workers are tasked with summarizing dialogue into discrete, actionable "facts" (e.g., "User is allergic to shellfish"). To maintain a clean database, the system employs semantic similarity to identify potential duplicates. When the AI's confidence in a match is low, it triggers a human-in-the-loop task to verify and consolidate the information.

Automated Retrieval and Siloing

To manage cognitive load and privacy, Chorus:Mnemonic uses Latent Dirichlet Allocation (LDA) or Correlated Topic Models to identify the active topic of a conversation. A hidden database of past facts is queried, and only the top 10 most relevant facts are surfaced to the current workers.

Figure 1: Comparison between a raw, multi-worker conversational stream (left) and a filtered, contextually relevant view (right) that reduces noise and preserves privacy.

This "siloing" effect ensures that no single worker gains a complete profile of the user, as they only see information relevant to the immediate context.

Discussion and Key Insights

The development of Chorus:Mnemonic yielded several key insights into the design of hybrid intelligence systems:

  • Separation of Concerns: Decoupling memory curation from active chatting significantly improves the quality and accuracy of recorded facts.
  • AI as a Librarian: The system leverages AI for the "big data" task of searching thousands of interactions, while humans provide the "understanding" necessary to summarize and verify relevance.
  • Structural Privacy: By limiting information exposure based on topical relevance, the system creates a built-in privacy guardrail that is more robust than simple access controls.

Evaluation and Results

While Chorus:Mnemonic is a work in progress, preliminary findings from the baseline Chorus system are promising. In manual tests, workers successfully utilized shared fact lists to inform responses in 80% of conversations, even without prior context. Additionally, processing conversation history in segments (10-20 messages) proved more cost-effective than maintaining continuous worker presence during long, intermittent interactions.

For future quantitative evaluation, the authors propose using standard Information Retrieval (IR) metrics, such as discounted cumulative gain, to measure the precision and utility of the facts surfaced by the topic-modeling algorithm.

Conclusion and Future Work

Chorus:Mnemonic represents a significant step toward persistent, privacy-preserving memory in crowd-powered systems. Future work will focus on validating the perceived helpfulness of surfaced facts in real-time and formally quantifying the reduction in "information spread" to prove privacy benefits. Additionally, the authors plan to implement worker governance mechanisms, such as down-voting, to ensure the long-term integrity of the collective memory.

Summary updated

Abstract

Maintaining consistency is a difficult challenge in crowd-powered systems in which constituent crowd workers may change over time. We discuss an initial outline for Chorus:Mnemonic, a system that augments the crowd's collective memory of a conversation by automatically recovering past knowledge based on topic, allowing the system to support consistent multi-session interactions. We present the design of the system itself, and discuss methods for testing its effectiveness. Our goal is to provide consistency between long interactions with crowd-powered conversational assistants by using AI to augment crowd workers.
BibTeX
@inproceedings{lasecki2013automated,
  author = {Walter Stephen Lasecki and Jeffrey Philip Bigham},
  title = {{Automated Support for Collective Memory of Conversational Interactions}},
  booktitle = {Hcomp (works in Progress / Demos)},
  year = {2013},
}

Loading related content...

Figure