TL;DR
This paper introduces OLKAVS, the largest publicly available Korean audio-visual speech dataset, featuring 1,150 hours of transcribed audio from over 1,100 speakers across nine viewpoints. The authors demonstrate its utility by releasing baseline models for audio-visual speech recognition (AVSR) and lip-reading, showing significant improvements with multi-modal and multi-view approaches.
What Problem Does This Paper Address?
Understanding human speech is a multi-modal process, combining auditory cues with visual information from lip movements and facial expressions. While numerous audio-visual speech datasets exist, they predominantly focus on English, often rely on pre-existing videos, and typically offer limited multi-view perspectives. This scarcity of large-scale, multi-view, non-English datasets hinders research in crucial areas like robust speech recognition in noisy environments and advanced lip-reading technologies. Specifically for Korean, existing datasets are too small to effectively train modern deep learning models.
Methods
The paper introduces the OLKAVS dataset, a significant undertaking in data collection. Key aspects of its creation include:
- Large Scale: 1,150 hours of transcribed audio from 1,107 Korean speakers.
- Multi-View Recording: Each speaker was recorded from nine different viewpoints simultaneously (five cross views: frontal, up, down, left, right; and four diagonal views: upper left, upper right, lower left, lower right).
- Diverse Scenarios: Recordings were conducted in a studio with six different noise conditions, including no noise, indoor ambiance, indoor noise, traffic noise, construction site noise, and natural outdoor noise.
- Comprehensive Transcription: All audio was manually transcribed, providing ground truth in Korean.
- Baseline Models: The authors developed and released pre-trained baseline models for two tasks:
- Audio-Visual Speech Recognition (AVSR): Using a conformer sequence-to-sequence architecture, they compared uni-modal (audio-only, visual-only) training against multi-modal (audio-visual) training.
- Multi-View Lip Reading: They evaluated the benefit of using all available views versus only the frontal view for lip-reading models.
Datasets
The primary contribution is the introduction of the Open Large-scale Korean Audio-Visual Speech (OLKAVS) dataset. This dataset is characterized by:
- Size: 1,150 hours of audio, 5,750 hours of video.
- Speakers: 1,107 Korean speakers, with a balanced distribution of age and gender.
- Views: Nine synchronized viewpoints.
- Content: Transcribed speech covering 14 different topics, with scripts averaging 2,750 characters.
- Noise Conditions: Six distinct noise scenarios.
The paper also references existing datasets for comparison and context, including OuluVS2, AVICAR, LRS2-BBC, LRS3-TED, MV-LRS, VoxCeleb2, GLips, MISP2021, and smaller Korean datasets like those by Jo et al. and Lee et al.
Key Insights and Ideas
- Multi-modal Advantage is Clear: The experiments strongly reinforce that combining audio and visual information significantly improves speech recognition performance, especially in noisy environments. The AV-model outperformed uni-modal models, demonstrating the power of fusion.
- Multi-View Matters for Lip Reading: Using multiple viewpoints for lip-reading training leads to a substantial improvement in accuracy compared to relying solely on frontal views. This highlights the richer information available from diverse perspectives.
- Korean Speech Research Needs Large Datasets: The paper underscores the inadequacy of previous Korean audio-visual speech datasets for modern deep learning, justifying the need for a large-scale resource like OLKAVS.
- Noise Robustness is Crucial: The inclusion of various noise conditions in OLKAVS is vital for developing speech recognition systems that perform reliably in real-world scenarios.
Key Results
- AVSR Performance: The AV-model showed a notable performance improvement over uni-modal models in noisy conditions, while performing comparably to the audio-only model in clean data. Specific metrics like CER, WER, and sWER are provided in Table 2, showing the AV-model's superiority.
- Lip Reading Performance: The "All-model" (using all views) achieved a 22% relative reduction in Character Error Rate (CER) compared to the "F-model" (frontal view only) on a noise-free subset, demonstrating the benefit of multi-view training. Table 3 shows CERs for different views, with centered views generally underperforming side views.
Limitations and Future Work
- Limitations:
- While the dataset is large, it is still a "studio setup" and may not fully capture the complexities of unconstrained, in-the-wild speech.
- The paper notes that some teenagers were only presented with a subset of topics, which might limit certain analyses.
- Future Work:
- The authors expect OLKAVS to facilitate research in broader areas beyond AVSR and lip reading, including Korean speech recognition, speaker recognition, and mouth motion analysis.
- The release of the dataset and baseline models is intended to encourage further research and development in these areas.
Summary updated