Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whisper

Chih-Kai Yang, Kuan-Po Huang, Hung-Yi Lee

Abstract

Whisper, OpenAI's state-of-the-art speech recognition model, utilizes prompts to guide transcription. This study investigates whether Whisper truly "understands" these prompts or merely reacts to them as arbitrary noise. By testing semantic and language consistency, the researchers find that Whisper often performs better with semantically incorrect prompts and that English prompts consistently outperform native-language prompts, even on non-English audio. These findings suggest a significant gap between human-expected prompt understanding and actual model behavior.

Introduction

Prompting is a ubiquitous technique for steering large-scale models. In Automatic Speech Recognition (ASR), users provide textual "hints"—such as "This is a talk about biology"—to improve accuracy. However, a fundamental domain shift exists: during training, Whisper was optimized to predict text based on actual preceding transcripts, whereas inference prompts are often custom-made. This research asks: Is Whisper comprehending the semantic meaning of these prompts, or is it simply being "activated" by specific patterns?

Methodology

The researchers developed a systematic framework to evaluate prompt understanding across three axes:

  1. Semantic Consistency: Comparing "matched" prompts (correct topic) with "mismatched" prompts (intentionally incorrect topic).
  2. Language Consistency: Comparing English prompts against native-language prompts (e.g., Mandarin).
  3. Special Token Manipulation: Testing how Whisper handles language tokens (e.g., <|en|>, <|zh|>) when one is intentionally incorrect.
Experimental framework for testing textual prompts and language tokens.

To quantify adherence, the study introduces the Topic-Following Rate (TFR): the percentage of instances where the model achieves its lowest error rate when provided with the correct topic prompt. If a model truly understands a prompt, the TFR should approach 100%.

Experimental Setup

The study utilized several multi-domain and code-switched datasets:

  • GigaSpeech: English ASR corpus categorized by topics (e.g., Arts, Sports).
  • ASCEND: A Mandarin-English code-switched corpus with diverse conversational topics.
  • CSZS (Code-Switching from Scratch): High-quality datasets for Mandarin-English and French-English speech.

Key Insights

The "Mismatched" Paradox

Counter-intuitively, Whisper often achieves better Word Error Rates (WER) when given a semantically incorrect prompt. In some instances, mismatched prompts yielded an 11% relative improvement over matched ones. This suggests that Whisper is not "understanding" the topic but is instead being triggered by certain text patterns.

English as a Universal Trigger

English prompts consistently outperformed Mandarin prompts, even for transcribing Mandarin audio. On the ASCEND-zh dataset, using English prompts instead of Mandarin ones led to a 15% relative improvement in performance. This is likely due to the vast disparity in English training data, making English text structures more effective "anchors" for the decoder.

Low Topic Adherence and Token Robustness

The TFR remained consistently below 30%, indicating poor semantic grounding. However, Whisper showed surprising robustness with special tokens. In code-switched scenarios, the model effectively "filtered" impossible language tokens (e.g., ignoring a French token for Mandarin-English audio) rather than blindly following them.

Results

The quantitative analysis reveals that manual prompting can be a double-edged sword. In many cases, providing any textual prompt resulted in worse performance than a "no-prompt" baseline, indicating that manual prompting can sometimes degrade accuracy.

Summary of performance metrics across datasets, including TFR and comparison to no-prompt baselines.

As shown in the table above, the relative improvement from mismatched prompts is notable across all datasets, particularly in GigaSpeech (11%).

Conclusion

Whisper does not "understand" prompts in a human-centric way. Its behavior is driven by pattern recognition and training data biases (favoring English). Future work should focus on automated prompt engineering to discover "machine-expected" patterns and improving the semantic grounding of future ASR models to ensure they respect user-provided constraints.

Summary updated

Abstract

This research explores how the information of prompts interacts with the high-performing speech recognition model, Whisper. We compare its performances when prompted by prompts with correct information and those corrupted with incorrect information. Our results unexpectedly show that Whisper may not understand the textual prompts in a human-expected way. Additionally, we find that performance improvement is not guaranteed even with stronger adherence to the topic information in textual prompts. It is also noted that English prompts generally outperform Mandarin ones on datasets of both languages, likely due to differences in training data distributions for these languages despite the mismatch with pre-training scenarios. Conversely, we discover that Whisper exhibits awareness of misleading information in language tokens by ignoring incorrect language tokens and focusing on the correct ones. In sum, We raise insightful questions about Whisper’s prompt understanding and reveal its counter-intuitive behaviors. We encourage further studies.
BibTeX
@misc{yang2024do,
  author = {Chih-Kai Yang and Kuan-Po Huang and Hung-Yi Lee},
  title = {{Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whisper}},
  year = {2024},
  eprint = {2406.05806},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.05806},
}
Figure