Repairing ASR output by Artificial Development and Ontology based Learning

C. Anantaram, Amit Sangroya, Mrinal Rawat, Aishwarya ChhabraIJCAI 2018 · DOI

TL;DR

This paper introduces a novel method to improve the accuracy of general-purpose Automatic Speech Recognition (ASR) systems, especially in domain-specific contexts. By combining bio-inspired "Artificial Development" (ArtDev) with domain ontologies and machine learning, the approach effectively "repairs" inaccurate ASR outputs, leading to significant accuracy gains.

What Problem Does This Paper Address?

General-purpose ASR systems, while impressive, often struggle when deployed in specialized domains. This is primarily due to a lack of domain-specific vocabulary, context, and training data. Consequently, ASR outputs can be inaccurate, hindering the effectiveness of domain-specific applications that rely on speech interfaces. Furthermore, environmental conditions and speaker accents can exacerbate these inaccuracies. Building custom ASR models for low-resource domains is also challenging due to data sparsity. This paper aims to bridge this gap by proposing a method to repair and improve the output of existing general-purpose ASR systems for domain-specific use cases.

Methods

The proposed method employs a multi-stage repair process inspired by biological development and learning:

  • Artificial Development (ArtDev) based Repair: This core idea treats an inaccurate ASR output as an "injured biological cell." The repair process involves using a domain ontology as the "genetic knowledge" of the domain.
  • Step 1: Matching Function: This step identifies potential matches between words in the ASR output and terms within the domain ontology. It uses a sliding window approach and considers both exact matches and "closeness" matches (phonetic and Levenshtein distance). Identified domain ontology entries (subject-predicate-object triples) are collected as candidate "genes."
    • Step 2: Mapping Function: This stage refines the matches by considering the context of domain terms. It checks if phonetically similar domain terms are more appropriate than the original ASR output words, replacing them if necessary.
  • Step 3: ML Repair (Ontology based Learning): A Long Short-Term Memory (LSTM) based machine learning model is trained on domain-specific sentences containing ontology terms. This model predicts the next/previous word given a substring. The model uses corrected substrings (marked with <ed> tags) from previous steps to predict and score potential replacements for unmarked words, further refining the sentence.
  • Step 4: Linguistic Repair: The final stage addresses remaining linguistic inconsistencies. This involves re-parsing the sentence, evaluating Part-of-Speech (POS) tags, and applying linguistic rules (e.g., ensuring correct determiner usage) or using external tools like "LanguageTool" to achieve a grammatically sound output.
  • Environment Factoring: A key aspect of the system is its ability to identify the speaking environment (e.g., noisy, silent, specific locations like a railway station). This environmental identification allows the system to select appropriate matching and mapping functions and thresholds from a pre-trained model, tailoring the repair process.

Datasets

The paper mentions using "publicly available datasets on ASR output of sentences from financial domain available on github (www.github.com/speech-corpus/asr_outputs)." This dataset consists of 500 sentences collected from YouTube videos of company financial quarter results.

Key Insights and Ideas

  • Bio-inspired Repair: The analogy of "Artificial Development" and "injured biological cells" provides an intuitive framework for understanding the repair process, drawing parallels between biological adaptation and AI system improvement.
  • Domain Ontology as "Genetic Knowledge": Leveraging domain ontologies as a source of "true genetic knowledge" is a powerful concept for guiding the repair process, ensuring that corrections are semantically relevant to the specific domain.
  • Multi-stage Refinement: The pipeline approach, moving from phonetic/lexical matching to contextual mapping, ML-based prediction, and finally linguistic correction, allows for a comprehensive and layered repair strategy.
  • Environment-Aware Adaptation: Recognizing that environmental factors significantly impact ASR performance and using this information to adapt the repair mechanism is a crucial practical consideration.

Key Results

The paper reports significant improvements in accuracy across all stages of the repair process. Table 1 shows the percentage improvement for different speakers and an overall improvement:

  • Overall Improvement: The combined repair process (Match + Map + ML + Ling) resulted in an average improvement of 9.01% in Word Recognition Rate (WRR).
  • Stage-wise Improvements: The individual stages also contributed to the overall gain, with the ML repair stage showing a substantial improvement of 4.90% on average.

Limitations and Future Work

  • Data Sparsity: The authors acknowledge that the ML model's performance was impacted by data sparsity, leading to a significantly worse perplexity on the testing data compared to the training data.
  • Domain Specificity: While the method is designed for domain-specific applications, the effectiveness might still depend on the richness and accuracy of the provided domain ontology.
  • Future Work: The paper suggests that further research could explore more sophisticated ways to incorporate environmental factors and potentially develop more robust ML models that are less susceptible to data sparsity. The authors also implicitly suggest that extending this approach to other domains beyond the financial one would be a natural next step.

Summary updated

BibTeX
@inproceedings{anantaram2018repairing,
  author = {C. Anantaram and Amit Sangroya and Mrinal Rawat and Aishwarya Chhabra},
  title = {{Repairing ASR output by Artificial Development and Ontology based Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/842},
}
Figure