Text-Adaptable Entity Tagging Inside a Factorized Speech Transducer

Speech recognizers produce "word soup": lowercase text with no punctuation, no sentence breaks, and no marking of which words are names, places, or dates. Systems that need entities, such as voice assistants, call analytics, or earnings-call search, usually pipe this soup into a text tagger, and the results are poor. A RoBERTa tagger that reaches 0.83 F1 on clean news falls to 0.37 on earnings-call transcripts. It misses 37% of entities even where the recognizer made no word errors, and it recovers only 18% of entities that contain a single misrecognized word (Why Aren't We NER Yet?). One fix is to tag entities inside the recognizer itself, as WhisperNER does. But that ties entity knowledge to the audio-trained network, so adding a new entity type or domain means synthesizing hundreds of thousands of text-to-speech clips (WhisperNER).

We propose to tag entities inside a factorized speech transducer. This is a recognizer whose language-model half can be retrained on text alone. Each emitted word gets an entity tag in the Begin/Inside/Outside (BIO) format. The tags are scored by a conditional random field (CRF), a sequence model that scores whole label sequences jointly rather than each label in isolation. The CRF score is deliberately split in two. A typed linguistic part is learned from text and can be retrained on text alone. An untyped acoustic part says only "a span starts here" or "a span continues here" and stays fixed. In two sentences: the recognizer decides where entity spans are partly from how the words sounded, and decides what type they are entirely from text. As a result, new entity types and new domains can be added with tagged text and no new audio.

Core Claim

The obstacle to adapting joint speech-and-entity models without new audio is acoustic-linguistic entanglement: in end-to-end recognizers, sound processing and word prediction share internal representations. Factorized neural transducers already untangle this for word prediction. They split the network into a "blank" predictor that handles timing alignment and a vocabulary predictor that acts as a plain language model, so text-only fine-tuning of the vocabulary predictor adapts recognition to a new domain (Factorized Neural Transducer; Improved FNT). Our claim is that the same split can be carried into a structured tag layer. The way to do it is to factor the tag emission score itself, not just the word predictor.

Let $w_{1:N}$ be a word hypothesis, $\tau_n$ the encoder frame at which word $n$ was emitted, $h_{\tau_n}$ the acoustic encoder state there, and $y_{1:N}$ a BIO tag sequence over entity types. The tag score is

$$ s(y \mid w, x) = \sum_{n=1}^{N} \Big[ e^{\text{ling}}_{y_n}(w_{1:N}) + e^{\text{ac}}_{\pi(y_n)}(h_{\tau_n}) + A_{y_{n-1}, y_n} \Big], \qquad p(y \mid w, x) = \frac{\exp s(y \mid w, x)}{\sum_{y'} \exp s(y' \mid w, x)}. $$

The terms are as follows. $e^{\text{ling}}$ is a typed emission computed by a small bidirectional text layer over vocabulary-predictor states, so it is trainable from text alone. $\pi$ maps a typed tag such as B-ORG to its untyped shape (B, I, or O). $e^{\text{ac}}$ is an untyped emission read from the acoustic encoder. $A$ is a transition matrix, with illegal moves such as O→I-X fixed at $-\infty$.

The design follows from the parameter split. Text-only adaptation updates $e^{\text{ling}}$, $A$, and the vocabulary predictor, and leaves the encoder, blank predictor, and $e^{\text{ac}}$ untouched. Because $e^{\text{ac}}$ never sees entity types, a type that appears only in adaptation text still benefits from acoustic span evidence such as pauses, pitch resets, and the slower delivery of names. This learned acoustic channel replaces the hand-built pause and pitch features that Dynamic CRFs fed to a post-hoc tagger.

Research Program

The work builds directly on prior components. The base recognizer is the improved factorized transducer (Improved FNT), used as is. The tag layer borrows the globally normalized labeling idea of Neural CRF Transducers, but only in first-order form, so exact Viterbi decoding still applies. That paper labels clean text tokens and has no notion of acoustic frames or blank alignment. Here, by contrast, each tag position is anchored to the frame where its word was emitted.

Training adds the CRF negative log-likelihood, conditioned on the reference words and their forced-alignment frames, to the transducer and language-model losses of the factorized model.

Decoding has two levels. The baseline level runs word beam search, then tag Viterbi over each n-best hypothesis. The tag-aware level carries a running CRF forward score within each word hypothesis and adds a weighted version of it to the beam score. Hypotheses that form well-shaped, plausible entity spans can then survive pruning. This is a principled replacement for the four-state heuristic beam of C-FNT. It is also the mechanism through which we hope to raise the 18% recovery rate for entities containing a misrecognized word. We will also run C-FNT's class-based dual path for long-tail names, generalized to all typed B- tags, as an optional add-on with its own ablation.

The analysis has three questions. How much entity F1 comes from the acoustic span emission, on in-domain data and after text-only adaptation? Does tag-aware beam search change which words are recognized inside entities? Does the factorization recreate the "modularity tax"? The original factorized transducer paid 8.32% vs 8.10% WER on LibriSpeech before adaptation (Factorized Neural Transducer), so the tag layer must be checked for the same cost.

Sentence boundaries are a secondary target. A second untyped chain over {none, comma, period, question} shares $e^{\text{ac}}$, mirroring the two-layer coupling of Dynamic CRFs. Its purpose is to test whether the acoustic channel also helps with the missing-boundary failures documented on spontaneous speech.

Baselines

The naive reference is the plain cascade: a factorized transducer adapted on target text for words only, followed by an off-the-shelf RoBERTa-Large BIO tagger trained on written text. This is the setup that Why Aren't We NER Yet? measured at 0.37 micro F1 on Earnings-21. On SLURP, the comparable pipeline figure is 70.84 entity SLU-F1 on noisy ASR versus 78.19 on gold text (SLURP).

The strongest cascade, which a reviewer will rightly insist on, uses the same adapted recognizer and a RoBERTa-Large CRF tagger fine-tuned on the same target-domain tagged text. That text is first made speech-like: lowercased, punctuation stripped, disfluencies injected, and passed through simulated recognition errors. No reported result found on Lacuna for this augmented cascade, so we will run it ourselves.

The strongest joint system is WhisperNER (WhisperNER). It reports 56.25 zero-shot F1 on VoxPopuli-NER against 53.55 for a Whisper-plus-GLiNER cascade. We will fine-tune it on our in-domain speech and adapt it to each target domain by synthesizing text-to-speech audio for exactly the adaptation text our model sees. We will also report the cost in GPU-hours and synthesized hours.

We add a shallow-fusion baseline. A factorized transducer decodes with an external LM trained on the target text, where the text carries inline span markers and internal-LM subtraction is applied (ILME). For long-tail names, we compare against C-FNT on its own Giga-name-test and NER-name-test. There C-FNT reports 72.6% and 70.4% person-name F1, against 55.6% and 52.2% for the plain factorized transducer (C-FNT).

Ablations isolate the key ingredients. The first removes $e^{\text{ac}}$, leaving a purely linguistic tag CRF on the vocabulary predictor. The second replaces the CRF with a per-word softmax under hard legality masking. The third makes the acoustic emission typed instead of untyped. The fourth turns off tag-aware beam search.

Every system gets the same in-domain speech, the same adaptation text, the same encoder size class, and the same hyperparameter search budget on the dev sets. Beating the field means beating the augmented cascade and the TTS-adapted WhisperNER on entity F1 after text-only adaptation, with confidence intervals that do not overlap across three seeds, without losing more than 0.2 absolute WER relative to the word-only factorized model.

Why not just…?

Why not just fine-tune a text tagger on speech-like target text and keep the cascade? This is the most serious threat. The 37% zero-error failure rate was measured with a tagger trained on clean written text, so it does not show that a tagger trained on lowercased, disfluent, error-injected text would also fail. We expect our model to keep two advantages. The first is acoustic boundary evidence that text cannot supply. The second is tag-aware decoding that can change misrecognized entity words; a downstream tagger can only label what the recognizer already emitted. Both predictions are tested directly. The first by the $e^{\text{ac}}$ ablation on the zero-word-error subset of SWNE and Earnings-21. The second by entity-word error rate inside gold spans. If the augmented cascade matches us on both, the honest conclusion is that augmentation, not architecture, is what matters, and we will report it that way.

Why not just adapt WhisperNER by synthesizing audio for the new text? It may work. Its inline markers already avoid the length blow-up that BIO tokens cause in autoregressive decoding. Our model avoids that blow-up a different way: the tag is a parallel label per emitted word, not extra output tokens. What WhisperNER cannot do is adapt without synthesis, and synthetic voices lack the prosody of real spontaneous speech. The matched-text comparison settles which approach adapts better. The cost comparison settles whether a tie still favors the text-only route.

Why not just put a softmax tag head on the vocabulary predictor and mask illegal transitions? Hard masking removes illegal sequences, which is the main documented benefit of CRF layers in Structured prediction for clinical text. It could therefore capture most of the gain. Our reason for keeping a CRF is that global normalization lets the fixed acoustic and adaptable linguistic emissions be traded off against learned transitions, and that the transitions themselves can be retrained on text. The masked-softmax ablation tests this. If it comes within noise of the full model after adaptation, we will drop the CRF.

Why not just fuse an external marker-tagged LM? Shallow fusion with internal-LM subtraction is cheap. However, factorized text adaptation already beat fusion on words, by up to 30.2% relative WER (Improved FNT), and fused markers carry no acoustic span evidence. The fusion baseline measures whether that advantage carries over to entities.

Proposed Experiments

The main experiment tests text-only addition of new entity types. We train on VoxPopuli-NER speech with its entity labels, plus GigaSpeech for word recognition. We then adapt using only the text of the SLURP training set, whose slot types (dates, place names, persons, and others) partly differ from VoxPopuli's, and test on SLURP audio using SLU-F1. A second regime targets spontaneous speech. We adapt on silver-tagged financial text and evaluate entity micro F1 on Earnings-21 and on SWNE, aligning hypotheses to references as in Why Aren't We NER Yet?. Results are reported separately for the zero-word-error subset and for spans containing substitutions. The long-tail check runs on the C-FNT name test sets. The boundary chain is checked on LibriSpeech-PC with Punctuation Error Rate, where the joint Conformer reports 29.24% PER on test-clean (LibriSpeech-PC).

Three outcomes would be decisive. First, on SLURP, entity types seen only in adaptation text must show a gain from $e^{\text{ac}}$ comparable to types seen with audio; this is the untyped-transfer claim, and if it fails the factorization idea is wrong. Second, on Earnings-21 and SWNE, the full model must beat the augmented cascade specifically on the zero-error subset and on substitution-containing spans; if the margin appears only in aggregate, the architecture is not doing what we claim. Third, word error rate must stay within the modularity-tax band of the word-only factorized model, and WhisperNER with matched TTS adaptation must not exceed our entity F1 at a fraction of the adaptation cost.