This paper introduces Pisets, a robust, three-stage speech-to-text pipeline designed to overcome the common pitfalls of OpenAI’s Whisper model—specifically its tendency to "hallucinate" text during silences and its relative inefficiency with long-form audio. By combining specialized filtering with advanced uncertainty modeling, Pisets provides a production-ready solution for complex acoustic environments like scientific lectures and interviews.
Addressing the Hallucination Problem
While Whisper has revolutionized speech recognition, it faces two major challenges in real-world applications:
- Hallucination Artifacts: When faced with long silences or non-speech noise, Whisper often "imagines" grammatically correct but contextually nonsensical text.
- Computational Inefficiency: Processing long audio files sequentially is slow, and standard Voice Activity Detection (VAD) often fails to accurately segment speech in reverberant or noisy settings.
The authors specifically target the Russian language, which often lacks the robust, production-ready pipelines available for English, especially for specialized domains like scientific lectures.
The Three-Stage Architecture
The authors propose a "three-component architecture" that acts as a transformation chain to ensure only high-quality speech segments reach the final transcription engine.
- Primary Recognition (Wav2Vec2): Instead of a simple energy-based VAD, the system uses Wav2Vec2 as a "VAD on steroids." It identifies speech boundaries with high precision. This component was trained using curriculum learning, starting with clean data and gradually introducing noisier, more complex audio.
- False Positive Filtering (AST): To further eliminate hallucinations, segments from the first stage are passed through an Audio Spectrogram Transformer (AST). This model acts as a signal classifier, filtering out non-speech fragments like coughs, chair scrapes, or background noise that might have bypassed the initial detection.
- Final Transcription (Whisper): Only the "cleaned" segments reach the Whisper model. The authors fine-tuned Whisper using Bayesian Invariant Risk Minimization (BIRM), a technique that helps the model generalize across different acoustic environments, such as varying microphones or room echoes.
Uncertainty Modeling
Beyond simple transcription, Pisets implements Uncertainty Modeling. By comparing the outputs of different stages and analyzing Whisper’s token probabilities, the system can highlight "dubious" parts of the transcript. The authors found that by marking just 5% of the words as "uncertain," they could capture 35% of all transcription errors. This significantly improves the user experience for human editors, allowing them to focus only on the most likely errors.
Key Results and Performance
The system was evaluated on a custom test set of seven 20-40 minute Russian scientific lectures recorded in real-world conditions. The Pisets architecture showed clear improvements over standard models in both accuracy and robustness.
- Accuracy: On the long audio lecture dataset, Pisets achieved a Word Error Rate (WER) of 0.1065, significantly outperforming WhisperX (0.1683).
- Robustness to Noise: In high-noise environments (SNR = 1 dB), the fine-tuned "Whisper-Podlodka-V3" model maintained a WER of 0.2119, whereas the standard Whisper-Large-V3 degraded to 0.2409.
- Speed: The "smart" chunking approach with Wav2Vec2 allows for parallel processing, making the system faster than standard sequential Whisper approaches (averaging 131.5s vs 136.4s for noised audio).
Conclusion and Future Directions
The success of Pisets demonstrates that Whisper's hallucinations are often a failure of "attention" to silence rather than a linguistic failure. By ensuring the model only sees actual speech, hallucinations are nearly eliminated. While the system still faces challenges with homophones and complex pragmatic context, future work aims to incorporate larger multimodal models (like Qwen-Audio) and expand support to other languages, including Bengali and Spanish.
Summary updated