A cross-lingual spoken content search system

Jitendra Ajmera, Ashish VermaInterspeech 2011

TL;DR

This paper introduces a novel approach to spoken content search for languages with limited resources, by leveraging acoustic models trained on a resource-rich "base" language (like Indian-English) and phonetic language models trained on the target language (like Gujarati). This cross-lingual system achieves promising search accuracy, enabling content search where traditional ASR systems are infeasible.

What Problem Does This Paper Address?

The explosion of audio content online and in various applications (like CRM systems and telephony) necessitates efficient ways to search and index this information. Traditional spoken term detection (STD) methods often rely on large-vocabulary continuous speech recognition (LVCSR) systems. However, for many languages, especially regional ones, the lack of sufficient transcribed audio data and dictionaries makes training robust LVCSR systems extremely difficult, if not impossible. This paper tackles this "resource-scarce" language problem by proposing a method to search spoken content in a target language using models trained on a different, resource-rich language.

Methods

The core of the proposed system is a cross-lingual phonetic recognition approach. Instead of training acoustic models (AMs) on the target language, they are trained on a "base" language (Indian-English in their experiments) for which ample training data exists. The system then uses these base-language AMs in conjunction with a phonetic language model (PLM) specifically trained for the target language (Gujarati).

Here's a breakdown of the key steps:

  1. Base Language AM Training: Acoustic models are trained on a resource-rich language (Indian-English) using standard toolkits.
  2. Target Language Text Processing: Text data from the target language (Gujarati) is collected and transliterated into the base language's script.
  3. Grapheme-to-Phoneme (G2P) Conversion: A G2P model, trained on the base language, is used to convert transliterated target language words into phonetic sequences. This is a crucial step for bridging the linguistic gap.
  4. Target Language PLM Training: A phonetic language model (PLM) is trained for the target language using the generated phonetic sequences.
  5. Phonetic Recognizer: The base language AMs and the target language PLM are combined to create a phonetic decoder. This decoder is used to generate phonetic lattices for the target language audio content.
  6. Indexing: The phonetic lattices are converted into weighted finite-state transducers (WFSTs) and indexed for efficient search.
  7. Search: Queries (text or spoken) are also converted into phonetic representations (WFSTs). A composition operation between the query WFST and the index WFST retrieves matching content.

Datasets

  • Base Language Training Data: 180 hours of Indian-English audio data with transcripts, from approximately 1500 speakers.
  • Base Language Evaluation Data: 8000 utterances of Indian-English audio data.
  • Target Language Text Resources: Approximately 1500 paragraphs collected from 20 online news resources in Gujarati.
  • Target Language Audio Data: 15 hours of spoken Gujarati content from a radio program, featuring discussions on farming topics.

Key Insights and Ideas

  • Leveraging Shared Acoustic Properties: The fundamental idea is that acoustic models trained on one language can generalize to some extent to phonetically similar languages, especially when the target language has overlaps with the base language.
  • Phonetic Language Models are Crucial: While acoustic models can be shared, a language-specific phonetic language model is essential for achieving good performance in the target language. An approximated PLM for the target language significantly outperforms using the base language PLM.
  • Cross-Lingual Phonetic Recognition is Feasible: The experiments demonstrate that it's possible to achieve reasonable search accuracy for a resource-scarce language (Gujarati) by using acoustic models from a resource-rich language (Indian-English).
  • Index Size Reduction: Using Indian-English AMs for Gujarati content resulted in a smaller index size compared to using Gujarati AMs (if they were available), suggesting potential efficiency gains.

Key Results

The paper reports the following key quantitative results:

  • P@10 (Precision at 10) for Gujarati content:
    • Using Indian-English AM and Gujarati PLM: 60%
    • Using Indian-English AM and Gujarati PLM, with lattice-cut optimization: 65%
  • Comparison: The proposed system with an approximated Gujarati PLM achieved significantly better results (60%) than using an Indian-English PLM (46%) for Gujarati content.
  • Index Size: The index for 15 hours of Gujarati data using Indian-English AMs was 1GB, while for Indian-English content of the same duration, it was 0.5GB.

Limitations and Future Work

  • Phoneme Set Overlap: While Indian-English has a larger phoneme set than American-English, there are still phonemes present in Gujarati that are not in the Indian-English phoneme set. Additionally, a significant percentage of tri-phones in Gujarati were not present in the base language.
  • Approximated PLM: The Gujarati PLM was approximated, and the authors believe a more direct training approach would yield even better results.
  • Textual Queries: The experiments primarily focused on textual queries. The authors plan to perform experiments with spoken queries.
  • Adaptation Mechanism: The authors suggest that incorporating a quick adaptation mechanism, possibly by using a small amount of transcribed regional language data to estimate a more reliable P2P confusion matrix, could further improve performance.

Summary updated

BibTeX
@inproceedings{ajmera2011cross,
  author = {Jitendra Ajmera and Ashish Verma},
  title = {{A cross-lingual spoken content search system}},
  booktitle = {Interspeech},
  year = {2011},
}
Figure