Lexical-phonetic automata for spoken utterance indexing and retrieval

Julien Fayolle, Murat Saraçlar, Fabienne Moreau, Christian Raymond, Guillaume GravierInterspeech 2012

TL;DR

This paper introduces a novel method for indexing and retrieving spoken content by building hybrid lexical-phonetic automata. By combining lexical and phonetic information with weighted transitions and a supervised filtering approach, they achieve improved spoken utterance retrieval, particularly for named entities.

What Problem Does This Paper Address?

Searching for specific spoken content within large audio archives, like news broadcasts, is challenging. Automatic Speech Recognition (ASR) systems, while improving, still produce transcripts with errors, especially for out-of-vocabulary (OOV) words and named entities (like people's names or locations). These errors make traditional text-based information retrieval (IR) methods less effective. Existing approaches either use separate lexical and phonetic indexes in a "cascade" (which can be inefficient) or try to combine them into a single "hybrid" index. This paper aims to improve the accuracy and recall of spoken utterance retrieval by proposing a more robust hybrid index and a sophisticated matching and filtering strategy.

Methods

The core of the proposed method is the construction of lexical-phonetic automata. These are weighted finite-state automata that can represent both lexical (word-level) and phonetic (sound-level) information simultaneously.

  1. Index Construction: For each spoken utterance, a lexical-phonetic automaton is built. This automaton's transitions are weighted by a vector of features, including edit distance scores (measuring how well a query matches the utterance at a symbol level) and confidence measures from the ASR system.
  2. Query Processing: The textual query is also converted into a lexical-phonetic automaton.
  3. Matching: A key innovation is the use of edit transducers to match the query automaton with the index automata. Three types are explored:
  4. Perfect Matching: Only allows exact matches.
  5. Imperfect Matching: Tolerates all phonetic errors (too greedy).
  6. Semi-Imperfect Matching: A novel approach that allows phonetic errors but within constraints (e.g., a certain rate of correct phonemes within a sliding window). This offers a good balance.
  7. Filtering: After matching, a list of candidate utterances is generated. A supervised learning approach (using decision trees) is employed to filter these candidates. This filter uses the feature vector (edit distances, confidence scores) to estimate the probability that an utterance contains the query, allowing for a more precise ranking and selection.

Datasets

The experiments were conducted on 6 hours of French radio broadcast news from the ESTER2 corpus. This corpus includes reference transcripts with manually annotated named entities. The ASR system used had a word error rate (WER) between 16.0% and 42.2% on this data. The data was automatically segmented into 3447 utterances.

Key Insights and Ideas

  • Complementarity of Lexical and Phonetic Information: The paper strongly demonstrates that lexical and phonetic representations are complementary. While the lexical level is good for correctly recognized words, the phonetic level is crucial for handling recognition errors and OOV words.
  • The Power of Hybrid Indexes: Hybrid indexes, which integrate both lexical and phonetic information, offer significant advantages over cascaded approaches, especially when dealing with imperfect ASR outputs.
  • Semi-Imperfect Matching is Key: The proposed "semi-imperfect" matching strategy, which balances strictness and flexibility, proves to be much more effective than perfect or overly permissive matching.
  • Supervised Filtering is Essential: Relying solely on confidence scores or edit distances isn't enough. A supervised learning approach that combines multiple features provides a more accurate way to rank and select relevant utterances.

Key Results

  • The proposed hybrid approach with semi-imperfect matching and supervised filtering (dt-all) consistently outperformed the baseline (cascaded search with perfect matching and threshold filtering) across various query types.
  • For IV (in-vocabulary) queries, the hybrid search achieved the best results.
  • For OOV (out-of-vocabulary) queries, hybrid, cascaded, and phonetic searches performed similarly, as they all primarily rely on phonetic information.
  • The hybrid search (with semi-imperfect matching and dt-all filtering) delivered the best overall spoken utterance retrieval performance.
  • The paper highlights that for named entities, lexical recognition was successful for 73.89% of lemmas, and for the mis-recognized ones, the phonetic level correctly identified 67.73% of phonemes, underscoring the need for both.

Limitations and Future Work

  • Ranking of Mixed Queries: The authors noted that for mixed IV/OOV queries, the hybrid search sometimes performed worse than cascaded or phonetic searches. They hypothesize this is because the current ranking might overemphasize lexical matches even when they are not highly relevant. They suggest incorporating a tf*idf score into the feature vector to address this.
  • Parameter Tuning: The parameters for the semi-imperfect matching (e.g., window size a and phoneme rate p) were arbitrarily set for preliminary experiments. Further tuning could potentially improve performance.
  • Future Directions: The authors implicitly suggest further exploration of feature engineering for the filtering stage and refining the ranking mechanism for different query types.

Summary updated

BibTeX
@inproceedings{fayolle2012lexical,
  author = {Julien Fayolle and Murat Saraçlar and Fabienne Moreau and Christian Raymond and Guillaume Gravier},
  title = {{Lexical-phonetic automata for spoken utterance indexing and retrieval}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure