TL;DR
The BIGOS V2 project introduces the most extensive and reproducible benchmark for Polish Automatic Speech Recognition (ASR) to date. By curating over 800 hours of speech data from 24 sources and evaluating 25 different models, this research provides a standardized "yardstick" for Polish ASR. It reveals that while commercial models like Whisper Large V3 lead the pack, spontaneous conversational speech remains a significant hurdle for the field.
What Problem Does This Paper Address?
Despite Polish being spoken by over 50 million people, it suffers from a "resource gap" in the AI world. While ASR models exist, the community lacks a standardized way to compare them. Existing datasets are often underutilized due to licensing hurdles or poor interoperability, and there is no unified toolchain to ensure that an evaluation performed by one researcher can be replicated by another.
Without a reliable leaderboard, it is difficult to track progress or understand how models handle the nuances of the Polish language—such as its complex grammar, varied accents, and the stark difference between formal "read" speech and messy, spontaneous conversation.
Methods
The author developed a three-stage system called BIGOS (Benchmark Intended Grouping of Open Speech):
- Survey & Catalog: A systematic review of existing Polish speech resources to identify high-quality, accessible data.
- Curation: A rigorous pipeline to "clean" the data. This included unifying audio formats (WAV 16-bit/16 kHz), normalizing amplitude, and standardizing text transcripts (UTF-8 encoding, removing redundant characters).
- Evaluation Toolchain: An open-source framework that manages the evaluation of 25 models across 9 ASR systems (including Google, Azure, OpenAI’s Whisper, and NVIDIA’s NeMo).
Crucially, the paper emphasizes normalization. ASR metrics can be wildly inconsistent if one model outputs "2024" and another "dwa tysiące dwudziesty czwarty." The BIGOS pipeline applies standardized normalization (lowercasing, punctuation removal, etc.) to ensure a fair "apples-to-apples" comparison.
Datasets
The paper introduces two primary curated meta-datasets drawn from 24 openly available sources, providing a diverse range of acoustic environments from professional studios to noisy field recordings:
- BIGOS V2: Focused on read speech. It aggregates 12 source corpora (like Common Voice, MLS, and Clarin Studio) totaling ~294 hours of audio from nearly 4,000 speakers.
- PELCRA for BIGOS: Focused on spontaneous speech. This is a more challenging set of ~530 hours of conversational Polish, including dialogues, interviews, and meetings.
Key Insights and Ideas
- The "Spontaneous" Tax: There is a massive performance gap between speech types. Models that achieve a respectable ~13% Word Error Rate (WER) on read speech often see that error rate double or triple (to ~30-34%) when faced with spontaneous, conversational Polish.
- Normalization is Non-Negotiable: Applying full text normalization reduced the WER by a staggering 14.63 percentage points on the BIGOS V2 dataset. Without standardized normalization, benchmark results are essentially meaningless.
- Commercial vs. Open Source: Commercial "paid" APIs (Azure, Whisper Cloud, AssemblyAI) generally outperform free models by about 2.5 to 4.5 percentage points in median WER. However, the gap is narrowing as open-source models like Whisper Large V3 show elite performance.
- Hallucination Risks: Manual inspection revealed that some models (particularly Whisper variants) tend to "hallucinate" text when faced with unnaturally slow speech or long silences, highlighting a specific failure mode for transformer-based ASR.
Key Results
- Top Performers: The three best models identified were Whisper Large V3, Whisper Cloud, and AssemblyAI best.
- Read Speech (BIGOS V2): Paid systems achieved a median WER of 12.96%, while free systems trailed at 15.47%.
- Spontaneous Speech (PELCRA): The difficulty spiked significantly, with paid systems hitting a median WER of 29.90% and free systems at 34.18%.
- Scaling Laws: As expected, larger models generally performed better, but the paper notes that efficient architectures like Nemo Multilang and Nemo Conformer punch above their weight class relative to their parameter count.
Limitations and Future Work
The author identifies several areas for improvement:
- Data Leakage: Since many models are trained on massive web-scale datasets (like Common Voice or MLS), there is a risk that the test data was seen during training, potentially inflating performance scores.
- Gold Standard Transcripts: The benchmark relies on existing transcripts which may contain their own errors. Future work should involve manual verification of a "gold standard" subset.
- Semantic Metrics: Current metrics (WER, CER) are purely lexical. The author suggests moving toward "semantically informed" metrics that don't penalize a model for using a synonym that doesn't change the meaning of the sentence.
- Demographic Bias: While the study looked at age and gender, more work is needed to ensure models perform equitably across all Polish dialects and socio-demographic groups.
Summary updated