Abstract
Automatic Speech Recognition (ASR) for classroom lectures is often hindered by high Word Error Rates (WER) and the inherent disfluencies of spontaneous speech. This paper proposes a novel method to enhance the readability of lecture transcripts by treating the transformation from spoken to written Japanese as a machine translation task. By integrating a confusion network—which represents multiple recognition hypotheses—into the translation framework, the system robustly handles ASR errors and disfluencies. Experimental results demonstrate significant improvements in both WER and punctuation accuracy compared to standard ASR outputs.
Introduction
Transcribing classroom lectures via ASR is notoriously difficult due to domain-specific terminology and spontaneous speech characteristics such as fillers, restarts, and repetitions. Even state-of-the-art systems often encounter WERs between 30% and 50%. Furthermore, raw ASR transcripts lack punctuation and contain "junk" words that impede human readability. While various readability tools exist, they are typically designed for clean manual transcripts and often fail when applied to noisy ASR output. This research addresses the need for a robust framework to transform error-prone "spoken-style" ASR results into polished "written-style" text.
Methodology
The researchers frame the readability problem as a Machine Translation (MT) task: translating from "Spoken Japanese" to "Written Japanese." The core innovation is the utilization of a Confusion Network rather than a single "best" ASR hypothesis.
As illustrated in the confusion network diagram, the system considers multiple competing word possibilities at each time segment. This allows the model to "vote" on the most likely written word based on a distribution of spoken hypotheses. The framework consists of three key components:
- Translation Model: A set of rules mapping disfluencies to null tokens (deletion) and mapping silences to punctuation or grammatical particles (insertion).
- Language Model: A tri-gram model trained on nine years of newspaper articles to ensure natural written-style output.
- Stack Decoding: A beam search algorithm that traverses the confusion network to find the word sequence that maximizes the combined probability of the translation and language models.
The system was evaluated using 70 minutes of lecture data from the Corpus of Japanese Lecture Contents (CJLC), with acoustic models trained on the Corpus of Spontaneous Japanese (CSJ).
Results and Discussion
The proposed method significantly outperformed baseline ASR results. When compared against manually paraphrased (clean) transcripts, the system achieved a WER of 53.7%, a notable improvement over the 62.6% WER of the raw 1-best ASR result.
As shown in the detailed results table, the use of the confusion network provides robustness; by summing over multiple hypotheses, the system can recover the correct word even if the top ASR guess was incorrect. The system also excelled in punctuation insertion, achieving an F-measure of 0.422 for periods, compared to 0.335 for raw ASR pauses.
However, the study revealed that automatically inserting Japanese grammatical particles (e.g., wa, ga) actually degraded performance due to the introduction of false positives. Subjective tests indicated that while human readers overwhelmingly preferred the readability of the proposed method's output, "understandability" occasionally suffered if the system mistakenly deleted content words it identified as disfluencies.
Conclusion
By leveraging the multi-hypothesis nature of confusion networks within an MT framework, this research provides a robust path toward making messy lecture transcripts more accessible. Future work will focus on more sophisticated discriminative models to handle complex grammar like particle insertion and ensuring the preservation of low-frequency, high-importance domain terminology.
Summary updated