Abstract
Integrating named entity recognition (NER) with automatic speech recognition (ASR) can significantly enhance transcription accuracy and informativeness. In this paper, we introduce WhisperNER, a novel model that allows joint speech transcription and entity recognition. WhisperNER supports open-type NER, enabling recognition of diverse and evolving entities at inference. Building on recent advancements in open NER research, we augment a large synthetic dataset with synthetic speech samples. This allows us to train WhisperNER on a large number of examples with diverse NER tags. During training, the model is prompted with NER labels and optimized to output the transcribed utterance along with the corresponding tagged entities. To evaluate WhisperNER, we generate synthetic speech for commonly used NER benchmarks and annotate existing ASR datasets with open NER tags. Our experiments demonstrate that WhisperNER outperforms natural baselines on both out-of-domain open type NER and supervised finetuning.
Introduction
Traditional information extraction from voice recordings typically relies on a pipeline approach: an Automatic Speech Recognition (ASR) model transcribes audio to text, which is then processed by a Named Entity Recognition (NER) model. This sequential architecture suffers from two primary flaws. First, error propagation occurs when ASR mistakes (e.g., mishearing a proper noun) make it impossible for the NER model to correctly identify the entity. Second, these systems are often rigid, limited to a fixed set of labels defined during training.
WhisperNER addresses these challenges by proposing a unified, end-to-end model capable of performing both ASR and open-domain NER in a single pass. By prompting the model with arbitrary entity types at inference time, WhisperNER achieves "open-type" recognition, allowing it to identify new categories—such as "Celestial Bodies" or "Medical Symptoms"—without retraining.
Methodology
WhisperNER is built upon the Whisper architecture, a transformer-based encoder-decoder model. The core innovation lies in its ability to process audio and a list of target entity types simultaneously through prompt-based decoding.
Key Technical Innovations
- Prompt-Based Decoding: At inference time, the user provides a list of entity types as a prompt to the decoder. The model then identifies these specific types within the audio stream.
- Efficient Tagging Scheme: Instead of the standard BIO (Beginning, Inside, Outside) tagging used in NLP—which significantly increases the number of tokens the model must generate—WhisperNER uses a start/end marker format (e.g.,
<occupation>astronaut<occupation>). This keeps the output sequence short and inference fast. - Negative Sampling: During training, the model is prompted with "negative" entity types—labels that do not appear in the current audio clip. This teaches the model to be discerning and prevents it from "hallucinating" entities just because they were in the prompt.
- Logit Modification: The authors introduced a way to tune the model's sensitivity without retraining. By adding a bias to the "start tag" token during decoding, users can adjust the balance between precision and recall.
Datasets and Evaluation
To overcome the scarcity of high-quality speech data with open NER tags, the researchers utilized a creative data augmentation strategy:
- Synthetic Training Data: They took a large-scale synthetic text dataset (NuNER) and used Text-to-Speech (TTS) to generate 350,000 audio samples.
- New Benchmarks: Existing ASR benchmarks (VoxPopuli, LibriSpeech, Fleurs) were re-annotated using an LLM-based pipeline to create specialized NER evaluation sets.
- Standard Benchmarks: The model was also tested on the MIT-Movie and MIT-Restaurant datasets, converted to speech via TTS.
Results and Discussion
WhisperNER consistently outperformed strong pipeline baselines where Whisper is paired with state-of-the-art text NER models like GLiNER. On the VoxPopuli-NER benchmark, WhisperNER achieved a zero-shot F1 score of 56.25, surpassing the pipeline baseline of 53.55. Crucially, the addition of the NER task did not degrade transcription quality; in many cases, the Word Error Rate (WER) remained stable or even slightly improved compared to the base Whisper model.
Efficiency and Scaling
A major finding was the efficiency of the chosen tagging format. The choice of tagging format is a major bottleneck for generative models. The start/end marker approach was found to be much more computationally efficient than BIO tagging while maintaining high accuracy.
Conclusion
WhisperNER represents a significant step toward unified speech and language understanding. By integrating open NER directly into the ASR process, the model reduces error propagation and provides a flexible, efficient tool for extracting information from speech. While the model shows impressive zero-shot capabilities, future work will focus on expanding multilingual pre-training and reducing the reliance on synthetic training data.
Summary updated