Analysis about Theoretical Foundations for Method to Enhancing ASR Performance using OCR Word Frequency Differences

Kyudan Jung, Nam Joon Kim, Hyun Gon Ryu, Hyuk-Jae Lee

Abstract

As large language models (LLMs) gain prominence, the demand for high-accuracy automatic speech recognition (ASR) has increased, particularly in academic and technical domains. Specialized terminology often leads to high error rates in general-purpose ASR models. This paper investigates a method to improve ASR performance by leveraging word frequency differences between general text and lecture slides extracted via Optical Character Recognition (OCR). We address mathematical inconsistencies in previous approaches and establish the Power Law (Zipf’s Law) as a theoretical foundation for weighting specialized terms, resulting in a 3.22% improvement in recognition accuracy.

The Challenge: Specialized Terminology and the "Plateau" Problem

Standard ASR systems are typically trained on general conversational datasets. Consequently, they struggle with "out-of-vocabulary" (OOV) or rare technical terms common in specialized lectures (e.g., medical or engineering courses). While using OCR to identify terms from presentation slides is a known strategy, previous mathematical models for weighting these terms were flawed.

Specifically, when a technical word found on a slide did not exist in the general-purpose reference dataset, the "Relative Frequency" (RF) calculation became undefined due to division by zero. To circumvent this, researchers often assigned an arbitrary maximum value to these words. This created a statistical "plateau" where the ASR could not distinguish between highly critical jargon and moderately relevant terms, reducing the system's overall reliability.

Figure (a): The plateau problem in previous RF calculations, where top-ranked terms are assigned identical maximum values.

Methodology: Refining Relative Frequency

The proposed method utilizes three primary metrics to identify and weight specialized terms:

  1. Normal Frequency (NF): Word frequency in general contexts, derived from the Google Web Trillion Word Dataset.
  2. Lecture Frequency (LF): Word frequency in specific lecture slides, captured via OCR at 30-second intervals.
  3. Relative Frequency (RF): The ratio $LF / NF$. A high RF indicates a word is contextually specific to the lecture.

To resolve the plateau issue, the authors introduced two refinements:

  • Method 1 (Handling Missing Words): Instead of zero, unknown word counts are replaced with the minimum count value found in the OCR dataset. Furthermore, the NF calculation is restricted to words actually present in the OCR results to ensure consistency.
  • Method 2 (Power Law Alignment): To align the data with linguistic theory, RF values are capped at a minimum of 1. This ensures that only words appearing more frequently in the lecture than in general text receive a boost in the ASR decoder.

Theoretical Foundation: The Power Law

A key contribution of this work is the validation that specialized terms follow a Power Law distribution ($RF(w) \propto \text{rank}_{RF}(w)^k$). By applying the refinements in Method 2, the authors successfully aligned the experimental data with this theoretical model. This alignment provides a rigorous mathematical justification for the weighting coefficients used in the ASR's beam search decoder.

Figure (c): The refined RF distribution aligned with the Power Law (red line), showing a consistent slope for specialized terms.

Key Results and Insights

The implementation of these refined frequency modules led to several significant outcomes:

  • Accuracy Gains: The system achieved an ASR performance boost of up to 3.22% in technical lecture environments.
  • Statistical Fit: Filtering for RF values $\ge 1$ improved the statistical fit (Residual Standard Error) from 0.6606 to 0.3401, confirming the validity of the Power Law approach.
  • Visual-Audio Synergy: The research reinforces that visual signals (OCR from slides) are a powerful tool for resolving audio ambiguities in educational settings.

Conclusion

By grounding the Relative Frequency method in the Power Law, this research provides both a performance boost and a solid theoretical framework for ASR enhancement. Future work will focus on the system's dependency on OCR quality and tuning hyperparameters across different academic domains to ensure broad applicability.

Summary updated

Abstract

As the interest in large language models grows, the importance of accuracy in automatic speech recognition has become more pronounced. This is especially true for lectures that include specialized terminology. In such cases, the success rate of traditional ASR models tends to be low, presenting a significant challenge. A method using the word frequency difference approach has been proposed to improve ASR performance for specialized terminology. We investigated this proposal through experiments and data analysis to determine if it effectively addresses the issue. In addition, we introduced the power law as the theoretical foundation for the relative frequency methodology mentioned in this approach.
BibTeX
@inproceedings{jung2025analysis,
  author = {Kyudan Jung and Nam Joon Kim and Hyun Gon Ryu and Hyuk-Jae Lee},
  title = {{Analysis about Theoretical Foundations for Method to Enhancing ASR Performance using OCR Word Frequency Differences}},
  booktitle = {ICCE},
  year = {2025},
  url = {https://openreview.net/forum?id=HRVrmTRlbC},
}
Figure