Real-time lecture transcription using ASR for Czech hearing impaired or deaf students

Petr Cerva, Jan Silovsky, Jindrich Zdansky, Jan Nouza, Jiri MalekInterspeech 2012

TL;DR

This paper presents a real-time lecture transcription system for Czech, designed to aid hearing-impaired students. It tackles the challenges of spontaneous Czech speech by developing domain-specific language models and employing unsupervised incremental speaker adaptation to improve accuracy on the fly.

What Problem Does This Paper Address?

Students with hearing impairments often struggle to fully participate in live university lectures. While existing systems offer post-lecture access to recorded content, they don't provide real-time support during the lecture itself. This paper addresses the need for a system that can convert spoken Czech lectures into text in real-time, making them accessible to hearing-impaired students in the classroom. The core challenges lie in the spontaneous nature of lecture speech, the inflective complexity of the Czech language, and the need for the system to adapt to individual speakers without extensive manual setup.

Methods

The system is built on a client-server architecture. The core is an Automatic Speech Recognition (ASR) module running on a server, processing an audio-video stream. Key innovations include:

  • Domain-Specific Language Modeling: Instead of a general language model, they created models specifically for economics and informatics lectures. This involved collecting and pre-processing large text corpora from various sources (broadcast news, theses, web discussions, verbatim lecture transcripts) and then using linear interpolation to combine them, optimizing weights based on perplexity.
  • Unsupervised Incremental Speaker Adaptation: To handle variations in speakers and acoustic conditions, the system adapts its parameters incrementally and without requiring explicit user enrollment. This involves a two-phase adaptation procedure using CMLLR (Constrained Maximum Likelihood Linear Regression) and MAP (Maximum A Posteriori) estimation on collected feature vectors and phonetic transcriptions. Adaptation occurs automatically every ten minutes of lecture time.
  • Text Post-Processing: The raw ASR output is further processed to improve readability. This module uses weighted Finite State Transducers (FSTs) to handle tasks like formatting digit strings, adding punctuation, and correcting capitalization. This post-processing is modular, allowing layers to be enabled or disabled.

Datasets

  • Training Data:
    • Verbatim transcripts of 10 economics lectures (83k words).
  • Verbatim transcripts of 6 informatics lectures (38k words).
  • A large training text database including broadcast news, verbatim lecture transcripts, and newspaper articles on economics and informatics.
  • 300 hours of speech recordings from 3940 male and 2030 female speakers for acoustic model training.
  • Evaluation Data:
    • 17 lectures (10 economics, 7 informatics) not included in the training data, recorded in varying quality and with different lecturers.

Key Insights and Ideas

  • Domain-specific LMs are crucial: Even with a general Czech ASR system, specialized language models significantly improve performance for lecture transcription.
  • Unsupervised adaptation is key for usability: The ability to adapt to speakers on-the-fly without manual intervention is essential for a practical real-time system.
  • Post-processing dramatically improves readability: Raw ASR output is often difficult to understand. A dedicated post-processing module is vital for making the transcripts usable for hearing-impaired individuals.
  • Incremental adaptation converges quickly: Significant improvements in Word Error Rate (WER) are achieved with just 10 minutes of adaptation data, with diminishing returns for longer adaptation periods.

Key Results

  • Language Model Improvement: Domain-specific LMs reduced perplexity significantly compared to a general LM. While the absolute WER reduction was modest (1.2%), the authors believe this could be improved with more training data.
  • Speaker Adaptation Improvement: Unsupervised incremental speaker adaptation reduced the Word Error Rate (WER) of a baseline system by 24% relatively, from 29.8% to 22.5%.
  • Adaptation Efficiency: A significant WER reduction was achieved with just 10 minutes of adaptation data. The two-phase adaptation process on 10 minutes of data takes approximately 15 seconds on a multi-core processor.
  • Post-processing Effectiveness: The post-processing module significantly improves the readability of the ASR output, making it more comprehensible.

Limitations and Future Work

  • WER Still Needs Improvement: While the system shows promise, the authors acknowledge that the reached WER is not yet satisfactory. They plan to collect more spontaneous data for acoustic and language model training.
  • Post-processing Enhancements: The positive response to the post-processing output motivates further improvements, including adding a layer for formatting simple mathematical terms and equations.
  • Language Support: The current system is for Czech. Future work could involve adapting the approach to other Slavic languages like Slovak or Polish.
  • Computational Complexity of Adaptation: While efficient, the two-phase adaptation process is computationally intensive, and further optimizations might be explored.

Summary updated

BibTeX
@inproceedings{cerva2012real,
  author = {Petr Cerva and Jan Silovsky and Jindrich Zdansky and Jan Nouza and Jiri Malek},
  title = {{Real-time lecture transcription using ASR for Czech hearing impaired or deaf students}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure