Abstract
Whisper, OpenAI's state-of-the-art speech recognition model, utilizes prompts to guide transcription. This study investigates whether Whisper truly "understands" these prompts or merely reacts to them as arbitrary noise. By testing semantic and language consistency, the researchers find that Whisper often performs better with semantically incorrect prompts and that English prompts consistently outperform native-language prompts, even on non-English audio. These findings suggest a significant gap between human-expected prompt understanding and actual model behavior.
Introduction
Prompting is a ubiquitous technique for steering large-scale models. In Automatic Speech Recognition (ASR), users provide textual "hints"—such as "This is a talk about biology"—to improve accuracy. However, a fundamental domain shift exists: during training, Whisper was optimized to predict text based on actual preceding transcripts, whereas inference prompts are often custom-made. This research asks: Is Whisper comprehending the semantic meaning of these prompts, or is it simply being "activated" by specific patterns?
Methodology
The researchers developed a systematic framework to evaluate prompt understanding across three axes:
- Semantic Consistency: Comparing "matched" prompts (correct topic) with "mismatched" prompts (intentionally incorrect topic).
- Language Consistency: Comparing English prompts against native-language prompts (e.g., Mandarin).
- Special Token Manipulation: Testing how Whisper handles language tokens (e.g.,
<|en|>,<|zh|>) when one is intentionally incorrect.
To quantify adherence, the study introduces the Topic-Following Rate (TFR): the percentage of instances where the model achieves its lowest error rate when provided with the correct topic prompt. If a model truly understands a prompt, the TFR should approach 100%.
Experimental Setup
The study utilized several multi-domain and code-switched datasets:
- GigaSpeech: English ASR corpus categorized by topics (e.g., Arts, Sports).
- ASCEND: A Mandarin-English code-switched corpus with diverse conversational topics.
- CSZS (Code-Switching from Scratch): High-quality datasets for Mandarin-English and French-English speech.
Key Insights
The "Mismatched" Paradox
Counter-intuitively, Whisper often achieves better Word Error Rates (WER) when given a semantically incorrect prompt. In some instances, mismatched prompts yielded an 11% relative improvement over matched ones. This suggests that Whisper is not "understanding" the topic but is instead being triggered by certain text patterns.
English as a Universal Trigger
English prompts consistently outperformed Mandarin prompts, even for transcribing Mandarin audio. On the ASCEND-zh dataset, using English prompts instead of Mandarin ones led to a 15% relative improvement in performance. This is likely due to the vast disparity in English training data, making English text structures more effective "anchors" for the decoder.
Low Topic Adherence and Token Robustness
The TFR remained consistently below 30%, indicating poor semantic grounding. However, Whisper showed surprising robustness with special tokens. In code-switched scenarios, the model effectively "filtered" impossible language tokens (e.g., ignoring a French token for Mandarin-English audio) rather than blindly following them.
Results
The quantitative analysis reveals that manual prompting can be a double-edged sword. In many cases, providing any textual prompt resulted in worse performance than a "no-prompt" baseline, indicating that manual prompting can sometimes degrade accuracy.
As shown in the table above, the relative improvement from mismatched prompts is notable across all datasets, particularly in GigaSpeech (11%).
Conclusion
Whisper does not "understand" prompts in a human-centric way. Its behavior is driven by pattern recognition and training data biases (favoring English). Future work should focus on automated prompt engineering to discover "machine-expected" patterns and improving the semantic grounding of future ASR models to ensure they respect user-provided constraints.
Summary updated