TL;DR
This paper introduces the Korpus tal-Malti Mitkellem (KMM), a new, dedicated corpus of spoken Maltese designed to be a "gold-standard" resource for language technology development and linguistic research. The KMM aims to address the scarcity of high-quality spoken data for Maltese by focusing on representativeness, balance, and robust annotation, with promising initial results from an automatic speech recognition (ASR) system.
What Problem Does This Paper Address?
Developing robust language technologies, particularly for automatic speech recognition (ASR) and text-to-speech (TTS) systems, heavily relies on the availability of large, high-quality spoken corpora. However, for low-resource languages like Maltese, such resources are scarce. Existing Maltese corpora are often text-based or consist of parliamentary speeches, which are stylistically distant from natural, everyday speech. This paper addresses the critical need for a dedicated, representative, and balanced spoken corpus for Maltese that can serve a wide range of users, from researchers to the general public, and facilitate advancements in Maltese language technology.
Methods
The KMM corpus is designed with a "Core collection" at its heart, intended to be a principled, extendable resource. The compilation process involves:
- Data Collection: A variety of text-types are collected, ranging from read speech (excerpts from novels) to unscripted data. Unscripted data includes tasks like explaining recipes, retelling stories, discussing topics, responding to prompts, and free speech. Both monologue (Mono) and dialogue (Dia) recordings are captured.
- Annotation: A multi-stage annotation process is employed. Initially, human annotators followed established SPAN Guidelines for Maltese. More recently, a Wav2Vec2-based ASR model (Wav2Vec2MT) has been used to generate first-pass transcriptions, which are then manually corrected by human annotators. This hybrid approach aims to improve efficiency and reduce human effort.
- ASR Evaluation: A key aspect of the work involves evaluating the performance of the Wav2Vec2MT model against human annotations. This is done by calculating Word Error Rates (WER) and Character Error Rates (CER) for both Mono and Dia data, and across different text-types.
Datasets
- Korpus tal-Malti Mitkellem (KMM) - Core Collection: This paper introduces the KMM Core collection, which, at the time of writing, comprises approximately 20 hours of spoken Maltese data. This is a newly compiled dataset specifically designed for this project.
- Existing Maltese Corpora (for context): The paper references existing Maltese corpora such as the Korpus Malti (v3.0 and v4.0), CommonLanguage, Common Voice, and MASRI-Headset, to highlight the existing landscape and the gap KMM aims to fill.
Key Insights and Ideas
- The "Gold-Standard" Approach: The KMM's design emphasizes creating a "gold-standard" corpus, prioritizing high audio quality and meticulous annotations. This is crucial for building reliable language technologies.
- Balancing Act: The project grapples with the inherent trade-offs in corpus design: representativeness, balance, homogeneity, and cost-effectiveness. The KMM aims to strike a balance, particularly by prioritizing unscripted data over purely read speech.
- Hybrid Annotation is Promising: The initial evaluation of using an ASR model (Wav2Vec2MT) for first-pass transcription, followed by human correction, shows promising results. This hybrid approach can significantly speed up the annotation process, making corpus development more feasible for low-resource languages.
- Dia Data is Challenging for ASR: The analysis reveals that dialogue (Dia) data, likely due to speech overlap, presents a significantly higher error rate for the ASR model compared to monologue (Mono) data.
Key Results
- Data Distribution: The KMM Core collection, as of the paper's reporting, has a strong skew towards unscripted data (85%), with a good balance between Mono (48%) and Dia (52%) speech.
- ASR Performance:
- Dia data exhibits a high Word Error Rate (WER) of 71.44%.
- Mono data shows significantly lower error rates, with the "Retell" task yielding the best performance (WER 22.68%, CER 8.25%).
- The Wav2Vec2MT model, fine-tuned on 50 hours of Maltese speech, is considered "robust" for Mono data, suggesting its output is already useful for annotation pipelines.
- Demographic Balance: The collected data shows a reasonable distribution in terms of sex, age, and education level, with ongoing efforts to improve female representation and recruit participants from diverse educational backgrounds.
Limitations and Future Work
- Annotation Time and Cost: The paper acknowledges that spoken corpus creation is inherently time-consuming and expensive, requiring careful consideration of cost-effectiveness.
- ASR Limitations: While promising, the ASR model still struggles with certain phonetic nuances, disfluencies, and especially with overlapping speech in dialogue. The error rates for Dia data are particularly high.
- Metadata Compilation: Metadata on speakers and recordings is still being compiled.
- Future Work:
- Continue expanding the Core collection, focusing on increasing free speech data.
- Further fine-tune the ASR model specifically on the KMM corpus to improve transcription accuracy, particularly for boundary placement and dialogue.
- Investigate diarization techniques to better handle multi-speaker recordings.
- Improve the CER/WER for standard ASR, which remains a challenge due to the limited amount of available Maltese data.
- Explore the use of the KMM for other language technology applications beyond ASR.
Summary updated