Review of the application of intelligent speech technology in education

Jianqing Gao, Genshun Wan, Kui Wu, Zhonghua FuJournal of China Computer-Assisted Language Learning 2022 · DOI

Abstract

Education is a primary application field for intelligent speech technology, encompassing automatic speech recognition (ASR) and speech assessment. This paper reviews popular application cases, including intelligent micro-lessons, online distance learning, and spoken language testing. While these technologies enable faster knowledge acquisition and more efficient language learning, three significant challenges remain: the recognition of domain-specific terms, the processing of code-switching (language mixing), and fine-grained mispronunciation diagnosis. This review discusses state-of-the-art solutions to these problems and explores the evolving roles of language teachers and researchers in this technical landscape.

Introduction

Intelligent speech technology has transitioned from a futuristic concept to a foundational tool in modern pedagogy. While general-purpose voice assistants like Siri or Alexa have become ubiquitous, the educational environment presents unique challenges that standard models often fail to address. In a classroom or testing setting, ASR systems must do more than just transcribe; they must act as pedagogical tools capable of objective scoring and diagnostic feedback. This paper explores the evolution of these technologies and the technical hurdles that must be cleared to reach their full potential.

Core Applications in Modern Education

The integration of speech technology into education is most visible in three areas:

  1. Automated Language Testing: Systems now handle massive-scale oral examinations. For example, the IFLYTEK system utilized during the 2014 Guangdong province college entrance exams scored over 600,000 candidates in just two days, demonstrating a level of robustness and objectivity that exceeds human capacity at that scale.
  2. Intelligent Micro-lessons and Distance Learning: Speech technology facilitates the creation of searchable, interactive video content and provides real-time transcription for online learners, making education more accessible.
  3. Computer-Assisted Language Learning (CALL): These tools allow students to practice pronunciation in a low-stakes environment, receiving immediate feedback without the "burden of listening" falling entirely on the teacher.

The Technical Evolution: From Modular to End-to-End

Historically, speech systems relied on complex, independent modules for acoustics, lexicons, and language modeling. The field has since shifted toward End-to-End (E2E) architectures, such as Connectionist Temporal Classification (CTC), RNN-Transducers (RNN-T), and Attention-based Encoder-Decoders.

While E2E models offer a more unified and efficient framework, they introduce a "black box" problem. Traditional models allowed for manual adjustments to the lexicon, but E2E models make it harder to inject specific "hot words" or technical terms. This trade-off between architectural simplicity and manual control is a central theme in current research.

Addressing the Three "Final Bosses" of Educational ASR

Standard ASR models often fail on specialized vocabulary. A student discussing biology might use the word "mitochondria," which a general model might misidentify. To solve this, researchers use Fusion Techniques (Shallow, Cold, and Component fusion). These methods allow for the injection of external domain knowledge into a pre-trained model without requiring a full retraining of the system. Specifically, Contextual Listen Attend Spell (CLAS) has proven effective at improving jargon recognition by biasing the model toward a specific set of keywords relevant to the subject matter.

2. Code-Switching

Language learners frequently mix their native tongue with the target language—a phenomenon known as code-switching. Training models to handle this is difficult due to a lack of naturally occurring data. Researchers have turned to Generative Adversarial Networks (GANs) to synthesize code-switching data (e.g., mixed Chinese-English sentences), providing the necessary training material to lower error rates in multilingual contexts.

3. Fine-Grained Mispronunciation Diagnosis

There is a significant difference between scoring (assigning a percentage) and diagnosis (identifying exactly why a sound was wrong). To achieve the latter, the paper suggests Multi-Context Modeling. This involves combining "long-context" models, which understand general intent, with "short-context" models that are hyper-sensitive to individual phoneme deviations. Unlike standard ASR, which is designed to be robust and "ignore" minor errors, diagnostic ASR must be sensitive enough to catch the tiny acoustic deviations that signal a learner's mistake.

Data Scarcity and Synthetic Solutions

Data sparsity remains a major hurdle, particularly for non-native speech. One innovative solution discussed is the use of Speech Synthesis (TTS) to generate "error data." By intentionally synthesizing common mispronunciations, researchers can train diagnostic models even when recordings of actual struggling students are unavailable.

Conclusion and Future Outlook

Intelligent speech technology is not intended to replace teachers but to augment them by removing the repetitive "burden of listening." While automated systems are already highly efficient at large-scale scoring, fine-grained diagnosis—the "holy grail" of CALL—is still evolving. Future work must focus on accent optimization, personalizing models to individual student speech patterns, and improving hardware integration to handle the noisy environments of real-world classrooms. As these technical barriers fall, the role of the educator will continue to shift toward high-level instructional design and personalized mentorship.

Summary updated

Abstract

Abstract Education is the main application field of intelligent speech technology, including automatic speech recognition and speech assessment. In this paper, we introduce some popular application cases of intelligent speech technology in education, such as intelligent micro-lessons, online distance learning and spoken English learning and testing. With the help of intelligent speech technology, people can acquire knowledge faster, learn languages more efficiently, and even communicate by using different languages. Moreover, we propose three kinds of challenging problems in the application of intelligent speech technology, including the recognition of domain-related terms, the recognition of code-switching and the fine-grained mispronunciation diagnosis. Viable state-of-art solutions to these problems are also discussed in detail. Meanwhile, we also discuss what language teachers and researchers can do in solving the above technical problems.
BibTeX
@article{gao2022review,
  author = {Jianqing Gao and Genshun Wan and Kui Wu and Zhonghua Fu},
  title = {{Review of the application of intelligent speech technology in education}},
  journal = {Journal of China Computer-Assisted Language Learning},
  year = {2022},
  doi = {10.1515/jccall-2022-0004},
}
Figure