New Dimensions in Testimony Demonstration

Ron Artstein, Alesia Gainer, Kallirroi Georgila, Anton Leuski, Ari Shapiro, David TraumNAACL 2016

TL;DR

This paper introduces "New Dimensions in Testimony," a novel dialogue system that allows users to converse with a persistent, pre-recorded representation of a Holocaust survivor. It addresses the challenge of preserving personal histories and enabling intimate connections with historical figures even when they are no longer alive, by simulating a live conversation using a large corpus of video statements.

What Problem Does This Paper Address?

The paper tackles the critical issue of preserving personal testimonies, particularly those of Holocaust survivors, for future generations. As the youngest survivors age, the opportunity for direct, personal engagement in educational settings diminishes. Traditional communication technologies either require real-time availability or lack the immediacy of a live conversation. While embodied conversational agents (virtual characters) have advanced, this work pioneers the use of such technology to create a time-offset, yet deeply personal, interaction with a real individual's recorded persona. The goal is to bridge the gap between historical record and personal connection, allowing future audiences to engage with survivor stories in an unprecedentedly intimate way.

Methods

The core of the "New Dimensions in Testimony" system is the NPCEditor, a component that combines Natural Language Understanding (NLU) and Dialogue Management. For NLU, it uses a classifier trained on question-response pairs to identify the most appropriate pre-recorded video statement from the survivor, Pinchas Gutter. Dialogue management handles situations where the classifier's confidence is low, employing an "off-topic" response strategy to either prompt the user for clarification or steer the conversation. The system leverages a large corpus of over 1700 recorded video responses (nearly 18 hours of content). To enhance the feeling of live presence, the system intersperses these responses with short videos of the survivor exhibiting idle behavior, which transitions to attentive listening when the user speaks. A mobile version of the system is also presented, adapting the experience for smaller screens and utilizing cloud streaming for video content.

Datasets

The paper does not explicitly detail the creation or size of the training dataset for the NPCEditor classifier beyond stating it's trained on "linked question-response pairs." The primary "dataset" is the extensive collection of pre-recorded video statements from Holocaust survivor Pinchas Gutter, totaling over 1700 responses and approximately 18 hours of video. This corpus was specifically curated for time-offset interaction.

Key Insights and Ideas

  • Bridging Time with Presence: The system demonstrates that by carefully curating and intelligently selecting pre-recorded video segments, a compelling illusion of a live, personal conversation can be created, even with a person who is not present in real-time.
  • The Power of Embodied, Time-Offset Interaction: This work highlights the potential of embodied conversational agent technology beyond fictional characters, extending it to preserve and share real human experiences in a deeply engaging manner.
  • Handling Uncertainty is Crucial: The emphasis on robust "off-topic" response strategies is key to maintaining conversational flow and user engagement when direct understanding fails, a common challenge in dialogue systems.
  • Scalability of Personal History Preservation: The technology offers a scalable solution for preserving the memories and stories of individuals, not just historical figures, but also ordinary people, family members, or experts.

Key Results

The system achieves a reasonable conversation flow, with approximately 64% of user questions receiving appropriate direct responses, 20% triggering off-topic responses, and 16% resulting in errors. The system's ability to handle noisy speech input and non-understanding is demonstrated in sample dialogues. The mobile version effectively reduces video file sizes by a factor of 16, enabling efficient streaming and a smaller app footprint.

Limitations and Future Work

The authors acknowledge that the system is a prototype and that the classifier's confidence threshold for direct responses is a critical parameter. Future work could involve further improvements to the NLU and dialogue management components to increase the percentage of direct responses and reduce errors. The mobile version's reliance on cloud streaming for most videos could be a limitation in areas with poor connectivity. The paper also implicitly suggests future work in expanding the corpus to include other individuals and exploring more sophisticated methods for dynamic response selection and generation.

Summary updated

BibTeX
@inproceedings{artstein2016new,
  author = {Ron Artstein and Alesia Gainer and Kallirroi Georgila and Anton Leuski and Ari Shapiro and David Traum},
  title = {{New Dimensions in Testimony Demonstration}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
}
Figure