Abstract
As interest in the Korean language grows globally, learners face significant challenges in mastering complex phonetics, such as double consonants and triphthongs. This paper proposes a novel system architecture for a Korean language learning application that utilizes a dual-track feedback mechanism. By integrating Google’s Speech-to-Text (STT) API for textual error detection with a Siamese deep learning network for acoustic similarity scoring, the system provides both granular word-level corrections and a high-level fluency score. Preliminary results demonstrate the system's ability to identify mispronunciations and visualize acoustic features for comparative analysis.
Introduction
The global rise of the "Korean Wave" (Hallyu) has led to a surge in demand for Korean language education, with enrollment in courses reaching record highs. However, Korean presents unique phonetic hurdles for non-native speakers, particularly regarding the distinction between liquid sounds (L/R) and the articulation of double consonants. While existing mobile applications focus heavily on vocabulary and grammar, there is a lack of specialized tools for automated, efficient pronunciation correction. Current solutions often fail to pinpoint specific errors or provide a quantitative measure of how closely a learner's speech matches a native speaker's patterns.
System Architecture and Methods
The proposed system is designed around a dual-track feedback loop that addresses both categorical and qualitative pronunciation errors. The architecture consists of three primary components:
1. Textual Feedback (Google STT API)
The user's voice is captured via a mobile interface and processed through the Google Cloud Speech-to-Text API. The system compares the resulting transcription against a target "correct" answer stored in the database. If a word is mispronounced significantly enough to be misidentified by the API, the application highlights the specific component in red, providing immediate visual reinforcement of the error.
2. Acoustic Feature Extraction
To analyze the qualitative aspects of speech, such as rhythm and intonation, the system uses the librosa library to transform audio signals into Mel-frequency cepstral coefficients (MFCCs). This process creates a visual representation (spectrogram) of the pronunciation, allowing the system to process audio as structured data.
3. Similarity Scoring (Siamese Network)
A Siamese deep learning network serves as the core technical innovation. This network consists of two identical sub-networks that process the user’s MFCC and a native speaker’s MFCC simultaneously. By calculating the L1 distance between these inputs and applying a sigmoid activation, the system generates a similarity score between 0 and 1. A score exceeding 0.9 is defined as "native-level" proficiency.
Preliminary Results and Proof of Concept
Initial testing utilized the AIhub emotional conversation dataset (KETI-02-002) to establish benchmarks. The researchers successfully demonstrated that the Google STT API could effectively identify intentional mispronunciations. For example, when a user provides an incorrect phonetic sequence, the system highlights the discrepancies against the target answer.
Furthermore, the system successfully visualized MFCC patterns for both correct and incorrect attempts, confirming that the Siamese network has distinct data signatures to utilize for similarity calculations. The 0.9 similarity threshold provides a clear, actionable goal for learners striving for fluency.
Discussion and Future Work
While the architectural design is robust, the project currently faces a significant hurdle in data scarcity. There is a lack of comprehensive datasets featuring non-native Korean speakers, which is essential for training the Siamese network to handle various foreign accents. Consequently, the system has not yet undergone large-scale user testing in a production environment.
Future research will focus on collecting diverse audio data from international learners to refine the network's accuracy. The ultimate goal is to deploy the application to evaluate whether this dual-feedback method—combining categorical text highlights with qualitative acoustic scores—leads to faster and more effective pronunciation improvement than traditional learning methods.
Summary updated