Investigating Cross-Language Speech Retrieval for a Spontaneous Conversational Speech Collection

Diana Inkpen, Muath Alzghool, Gareth Jones, Douglas OardNAACL 2006

TL;DR

This paper tackles the challenge of cross-language speech retrieval (CL-SR) on spontaneous conversational speech, a notoriously difficult task due to noisy transcriptions and translation issues. They demonstrate that by carefully selecting term weighting schemes and leveraging both automatic and manual metadata, significant improvements in retrieval accuracy can be achieved, even outperforming established IR systems.

What Problem Does This Paper Address?

Retrieving information from spoken audio, especially across different languages, is a complex problem. Unlike well-structured text documents, spontaneous speech is often messy: Automatic Speech Recognition (ASR) systems produce errors, and translating queries into the language of the audio introduces further inaccuracies. This paper focuses on the CLEF 2005 Cross-Language Speech Retrieval (CL-SR) task, which uses a collection of oral testimonies from Holocaust survivors. This dataset is particularly challenging because it consists of conversational speech with unclear topic boundaries, making it harder to pinpoint relevant segments compared to news archives. The goal is to build a system that can effectively search this spoken audio collection using queries in different languages.

Methods

The authors built their system using off-the-shelf Information Retrieval (IR) components. A key aspect of their approach involved:

  • Multi-System Machine Translation (MT): For cross-language queries, they translated topic statements into English using multiple free online MT tools and merged their outputs to capture a wider range of lexical choices.
  • Term Weighting Scheme Optimization: They extensively experimented with various term weighting schemes within the SMART IR system, using a set of training topics to identify the most effective combination for this specific collection. This involved exploring different ways to balance term frequency, document frequency, and collection frequency.
  • Phonetic Transcription: To mitigate ASR errors, they explored transforming text into phonetic transcriptions and indexing these phonetic n-grams. This approach aims to make the retrieval system more robust to speech recognition inaccuracies.
  • Metadata Integration: They investigated the impact of combining automatically generated metadata (from ASR and kNN classifiers) with manually assigned metadata (thesaurus terms and summaries).

Datasets

  • CLEF 2005 CL-SR Collection: This is the primary dataset used for evaluation. It comprises 8,104 manually identified, topically coherent segments from 272 interviews, totaling 589 hours of spontaneous conversational speech. The collection includes ASR transcripts (with a reported 38% word error rate), automatically assigned thesaurus terms (AK1, AK2), manually assigned thesaurus terms (MK), and 3-sentence summaries.
  • Training Topics: 38 topics were used to tune the system, particularly for selecting the optimal term weighting scheme.
  • Test Topics: 25 topics were used for final evaluation. These topics were structured into Title, Description, and Narrative fields and translated into Czech, German, French, and Spanish.

Key Insights and Ideas

  • Term Weighting is Crucial: The choice of term weighting scheme has a significant impact on retrieval performance, especially for shorter documents and smaller collections like the one used here. A carefully selected scheme (mpc.ntn) proved to be substantially better than many others.
  • Phonetics Help, But Don't Replace Text: Using phonetic n-grams as an indexing strategy can improve retrieval accuracy by compensating for ASR errors, but combining them with text-based indexing offers only marginal additional benefits.
  • Manual Metadata is Gold: Integrating manually assigned metadata (like thesaurus terms and summaries) with automatic descriptions dramatically boosts retrieval effectiveness, highlighting the enduring value of human annotation.
  • Cross-Language Performance Varies: While cross-language retrieval for Spanish and French showed comparable or even better results than monolingual English, German and Czech performance lagged, possibly due to translation quality issues and limited translation resources.

Key Results

  • The system developed by the authors achieved the best results among all seven participating teams in the CLEF 2005 CL-SR track.
  • For English topics using only automatic data, the best Mean Average Precision (MAP) achieved was 0.2176.
  • When incorporating manual metadata (MK, summaries, ASR transcripts, AK1, and AK2), the best MAP score for English topics reached 0.4647, a significant improvement.
  • Their system outperformed the University of Maryland's InQuery system (a widely used IR system optimized for large collections) by 6% relative for English TD and 10% relative for French TD when using only automated fields.

Limitations and Future Work

  • ASR Error Correction: The authors acknowledge that ASR errors remain a significant challenge. Future work will focus on investigating methods to remove or correct these errors in ASR transcripts, potentially using semantic coherence measures.
  • Understanding Weighting Effectiveness: They plan to further explore the relationship between collection properties and term weighting schemes to better understand why their chosen scheme (mpc.ntn) was so effective.
  • Expanding to New Collections: Future CL-SR tasks will involve new collections, including Czech interviews, which will present new challenges, particularly regarding unknown topic boundary conditions for ground truth relevance judgments.

Summary updated

BibTeX
@inproceedings{inkpen2006investigating,
  author = {Diana Inkpen and Muath Alzghool and Gareth Jones and Douglas Oard},
  title = {{Investigating Cross-Language Speech Retrieval for a Spontaneous Conversational Speech Collection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure