Multimodal Dialog with the MALACH Audiovisual Archive

Adam Chýlek, Luboš Šmídl, Jan ŠvecInterspeech 2019

TL;DR

This paper presents a multimodal dialog system for searching the MALACH Holocaust testimonies archive. It allows users to ask natural language queries in English and Czech, retrieving relevant video footage and context, making a vast historical resource more accessible.

What Problem Does This Paper Address?

The MALACH archive, containing thousands of hours of Holocaust testimonies, is largely untranscribed, with only hand-picked keywords for annotation. This severely limits search capabilities, making it difficult to find specific information within the vast collection. While previous work enabled full-text search on transcribed portions, the authors aim to bridge the gap for the untranscribed majority by enabling natural language dialog-based retrieval, allowing users to interact with the archive in a more intuitive and comprehensive way.

Methods

The system is built around two core components: a knowledge base (KB) construction pipeline and a dialog system.

  • Knowledge Base Construction: This is an offline process that leverages automatic speech recognition (ASR) to transcribe audio. Subsequently, a semantic entity detection (SED) approach, using context-free grammars, extracts knowledge about entities like cities, countries, dates, names, family members, and life events. These grammars are designed to be language-specific but extensible.
  • Dialog System: This system uses an agent-based dialog manager and is built on an in-house SpeechCloud framework, providing real-time ASR, SED, and text-to-speech (TTS). The dialog manager uses the same SED approach for natural language understanding as the KB construction, with additional dialog-specific entities. When a user query is processed, the system attempts to construct a KB search request. If no specific entities are found, it falls back to a full-text search on the ASR output. The system also incorporates features like voice activity detection for seamless interaction and pauses recognition during TTS output.

Datasets

The primary dataset is the MALACH archive of Holocaust testimonies. The paper does not mention specific datasets used for training the ASR or SED models, but implies that the system is trained on English and Czech data for these components.

Key Insights and Ideas

  • Bridging the Transcription Gap: The core innovation is enabling natural language search on a largely untranscribed audiovisual archive by intelligently extracting semantic information.
  • Multimodal Interaction: The system combines spoken language queries with a graphical user interface that displays video footage, transcripts, and metadata, offering a rich user experience.
  • Dialog for Exploration: The dialog system is designed not just for direct answers but for exploratory search, allowing users to follow up on information and discover related content even if their initial query wasn't explicitly present. For instance, if a witness doesn't mention a specific place, the system can suggest other witnesses who did.
  • Language Extensibility: The modular design, particularly the use of language-specific grammars and ASR training, makes it relatively straightforward to extend the system to other languages.
  • Seamless Fallback: The system gracefully handles queries that don't map to semantic entities by falling back to a full-text search, presenting it to the user as a unified search experience.

Key Results

The paper focuses on the system's functionality and design rather than presenting quantitative performance metrics. However, it demonstrates:

  • Successful implementation of a multimodal dialog system for searching the MALACH archive in both English and Czech.
  • The ability to retrieve relevant video segments and contextual information based on natural language queries.
  • A user interface that allows for exploration of search results, including video playback with highlighted mentions and metadata.
  • The system's capacity to handle follow-up questions and guide users through the archive's content.

Limitations and Future Work

  • Limitations: The paper acknowledges that a large portion of the archive is not manually transcribed, which is the primary challenge the system aims to address. While the system uses ASR, the accuracy of the ASR and SED components directly impacts the quality of the retrieved information. The paper does not provide specific accuracy metrics for these components.
  • Future Work: The authors plan to enhance the KB by deriving knowledge from external sources like web pages and other Holocaust knowledge bases. They also aim to improve the methods for knowledge extraction from transcripts.

Summary updated

BibTeX
@inproceedings{chylek2019multimodal,
  author = {Adam Chýlek and Luboš Šmídl and Jan Švec},
  title = {{Multimodal Dialog with the MALACH Audiovisual Archive}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure