Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency

Shigeki Karita, Richard Sproat, Haruko IshikawaACL 2023

TL;DR

This paper introduces a novel "lenient" evaluation metric for Japanese Automatic Speech Recognition (ASR) that accounts for the rampant and natural spelling variations in the language. By constructing a lattice of plausible alternative spellings, their method achieves a significant reduction in character error rate (CER), better reflecting real-world ASR performance.

What Problem Does This Paper Address?

Traditional Automatic Speech Recognition (ASR) evaluation relies on metrics like Word Error Rate (WER) and Character Error Rate (CER). While effective for languages with standardized spellings, these metrics struggle with languages like Japanese, which exhibit extensive orthographic variation. For instance, a single word can be written in hiragana, katakana, or kanji, and even within these scripts, multiple valid spellings can exist. This inconsistency means that a perfectly understandable transcription might be penalized as an error simply due to a different, yet valid, spelling choice. This paper aims to bridge this gap by proposing a more forgiving evaluation metric that acknowledges these natural variations, thereby providing a more accurate assessment of ASR systems for Japanese.

Methods

The core of the proposed method is the creation of a "lattice" of possible correct spellings for a given reference transcription. This lattice is constructed in several stages:

  1. Kana Conversion: Kanji sequences in the reference transcription are converted to their hiragana equivalents using a text normalization system. This step also implicitly provides word segmentation.
  2. Katakana Conversion: Hiragana words are also converted to their katakana counterparts, often used for emphasis.
  3. Kanji Restoration: A Neural Machine Translation (NMT) model is trained to convert kana spellings back into plausible kanji spellings. This is crucial because simply converting to kana might lose nuances or fail to capture common kanji usage. The NMT model is trained on web text where kana spellings were explicitly tagged for conversion.
  4. Lexicon Augmentation: To capture "safe" spelling equivalences (where one spelling can be substituted for another without changing meaning), the system leverages lexical resources like the Japanese Orthographic Dictionary. This adds further valid variants to the lattice.

The final lattice represents a rich set of plausible transcriptions. Evaluation then involves computing an edit distance between the ASR hypothesis and this lattice, rather than a single reference string.

Datasets

The paper does not explicitly mention newly introduced benchmarks. However, they evaluate their method on proprietary Japanese datasets from three domains:

  • Farfield: Likely involves speech captured from a distance.
  • Voice-Search (VS): Speech data from voice search queries.
  • YouTube (YT): Speech data from YouTube video segments.

These datasets contain anonymized and hand-transcribed utterances. For training, a large-scale multi-domain dataset of 25,000 hours of speech was used, with a specific YT domain model trained on 2,000 hours.

Key Insights and Ideas

  • Japanese Spelling is Fundamentally Ambiguous: Unlike English, where spelling variation is often minor or regional, Japanese orthography is inherently flexible, making standard evaluation metrics insufficient.
  • Modeling Variation is Key: The paper argues that ignoring spelling variation is akin to ignoring a fundamental aspect of the language. Their approach models this variation rather than trying to eliminate it.
  • Kanji-Kana Interplay is Crucial: The ability to convert between kana and kanji, and to select the most contextually appropriate kanji, is vital for accurate evaluation. The NMT model plays a key role here.
  • "Safe" Lexical Equivalences: The use of dictionaries to identify interchangeable spellings (e.g., different kanji for the same word with the same meaning) significantly expands the set of valid alternatives.
  • Lenient Evaluation Reflects Real-World Usability: A system that produces a valid, albeit different, spelling should not be penalized. This lenient approach better reflects how humans would perceive the accuracy of an ASR system.

Key Results

The proposed lenient evaluation method demonstrated a significant reduction in Character Error Rate (CER):

  • Absolute CER Reduction: Ranged from 2.4% to 3.1% across different domains.
  • Relative Error Reduction: Achieved up to 25.16% in the Voice Search domain.
  • Manual Evaluation: Human raters found that 95.4% of the generated spelling variants were plausible, with 16% being considered "great" or better than the original transcripts.

Limitations and Future Work

  • Proprietary Resources: The evaluation system relies on proprietary lexical resources and text normalization tools, which could limit its direct replication. The authors suggest that open-source alternatives like MeCab might be usable but could degrade performance.
  • Style and Register: The current method doesn't fully account for style and register, which can influence the appropriateness of certain spellings (e.g., formal vs. informal contexts). Future work aims to incorporate style/register language models to rank alternative spellings.
  • ASR Training: The current approach focuses on evaluation. Future work will explore using these augmented lattices during ASR system training to make models more robust to spelling variation.
  • Large Language Models (LLMs): The authors are investigating whether LLMs like GPT-4 or Bard can be used to judge the interchangeability of Japanese spellings.
  • Broader Applicability: While focused on Japanese, the approach is suggested to be applicable to other languages with significant spelling variation, especially those lacking a standardized written form.

Summary updated

BibTeX
@inproceedings{karita2023lenient,
  author = {Shigeki Karita and Richard Sproat and Haruko Ishikawa},
  title = {{Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure