Where Do Discrete Speech Tokens Lose the Question Mark?

Speech language models increasingly read audio as discrete tokens. A tokenizer maps each short slice of sound to one entry in a fixed vocabulary, much as text is split into subwords. The tokens are cheap and align well with language models. They also throw information away. When such a model rewrites a raw transcript into readable text, adding punctuation, capitalization and sentence breaks, some of what it needs lives only in the voice. Examples are the rising pitch that marks a Spanish question with statement word order, the pause before a new sentence, and the lengthened final syllable of a trailing-off clause.

This note proposes to find out where those cues die. We compare three things on the same utterances. The first is what human listeners can still hear in audio rebuilt from the tokens. The second is what a small probe can still decode from the tokens. The third is what a downstream transcript-formatting model actually uses. If listeners and probes still recover a boundary that the model misses, the loss lies in the model, which is ignoring the audio. If listeners cannot hear it either, the loss lies in the tokenizer. The two cases call for different fixes. The field currently treats them as one problem.

Core Claim

The claim has two parts, and both can be tested. First, the damage that discrete tokenization does to transcript normalization is concentrated in a few boundary cues, and different parts of the tokenizer destroy different cues. Collapsing repeated tokens, which is standard for self-supervised tokens, should mainly erase pause and final-lengthening information. Coarse quantization should mainly flatten terminal pitch movement. Second, the cues that matter are the ones human listeners use when they judge boundaries from prosody alone. Measuring how well each cue survives, word by word at word ends, therefore predicts downstream punctuation errors better than holistic codec quality scores such as PESQ or UTMOS do. If both parts hold, the targeted repair is to keep a handful of word-end numbers alongside the tokens, not to widen the whole codebook.

Grounding and Prior Work

Four lines of work shape this proposal.

The first supplies the need. Two papers show that punctuating spoken transcripts needs prosody. Leveraging Prosody for Punctuation Prediction of Spontaneous Speech adds word-end pitch, energy and pause features to BERT on Switchboard. Its gains are largest on noisy recognizer output and on incomplete sentences. Resolving Transcription Ambiguity in Spanish fuses an acoustic question detector with a lexical model and raises question-mark F1 by up to 4.4% over text alone. Both use continuous hand-crafted or filterbank features. Neither asks what happens when the audio arrives as discrete tokens, which is the setting we study.

The second supplies the loss. Speech Discrete Tokens or Continuous Features? runs a controlled comparison inside speech language models. Continuous features win on 5 of 6 tasks, and emotion recognition falls from 65.87% to 36.12% with discrete tokens. Discrete tokens win on phoneme recognition and train much faster. Emotion-Aware Quantization and AffectCodec show that the first few residual quantization layers carry most of the affective and prosodic information. Both then design codecs to preserve emotion. These studies measure emotion and utterance-level classes, not word-boundary structure. They also do not separate the quantizer from the collapsing of repeated tokens. That separation matters because Decoupling Segmental and Prosodic Cues through Vector Quantization used quantization plus duplicate removal precisely to strip timing and prosody from phonetic content. It paid a 0.20 drop in mean opinion score for naturalness. Their tool for removing prosody is, in our setting, the suspected culprit.

The third explains why a model can fail even when the cue survives. Heard but Not Heeded finds paralinguistic style decodable at 82–85% in upper encoder layers but only 19–53% at the output. Do Audio LLMs Listen or Read? shows that models side with the text when text and voice conflict. It partly repairs this by routing intermediate encoder layers to the language model. Neither work looks at tokenized input or at transcript formatting. Their findings are why we cannot infer tokenizer loss from downstream error alone.

The fourth supplies the measurement lens, from acoustic-perceptual correlation. Munson shows that crowdsourced mean ratings track acoustic properties reliably, which licenses online listening tests. Xue et al. warn that holistic quality judgments diverge from what listeners extract for lexical decoding. That warning is the basis of our prediction that PESQ and UTMOS will be poor proxies for boundary preservation. Gili Fivela et al. show that acoustic–perceptual correlations depend on stimulus complexity, so we stratify results by utterance length and boundary type.

Research Program

We build a three-level audit and apply it to two families of speech tokens. The first family is codec tokens, where we vary how many residual quantization layers are kept. AffectCodec is one codec in this family, alongside a standard codec of the EnCodec type. The second family is k-means clusters of HuBERT and WavLM features, where we vary cluster count and switch duplicate removal on and off.

At the perceptual level, we rebuild audio from codec tokens at each depth and low-pass filter it, which removes the words but keeps pitch and timing. Crowdsourced listeners then classify each word-final position as question, sentence end, trailing incomplete, interruption, or continuation. The same listening test on the original filtered audio sets the ceiling. Agreement between listeners per mark also tells us how acoustically marked each boundary type is. This speaks directly to the unresolved finding by Zhu et al. that adding comma and period tokens to the speech recognizer's output raised its word error rate by up to 4.8%. Our hypothesis is that these marks have weak and inconsistent acoustic signatures, so listener agreement on them will be low even on original audio.

At the representational level, a small probe predicts the same boundary labels from a window of tokens around each word end. It also predicts continuous cue values from the tokens: terminal pitch slope, pause duration, and final-syllable lengthening. The probe works for both token families, which covers the self-supervised tokens that cannot easily be turned back into audio.

At the use level, the same language model rewrites the transcript under several conditions: text only, text plus tokens, and text plus continuous features.

From these we compute two quantities per cue and configuration. Cue retention is the correlation between the probe-decoded cue and the cue measured on the original audio. Recoverable-but-unused mass is the share of boundaries that listeners or probes get right but the model gets wrong. We then test whether cue retention predicts listener loss and model F1 loss, and compare it against PESQ, UTMOS and codebook entropy as predictors. Only after this diagnosis do we test repairs. If the loss is in the tokenizer, we attach a few word-end numbers (pitch slope, pause, lengthening) as extra tokens. If the loss is in the model, we route intermediate encoder features or tokens to the model in the style of Pang et al.

Baselines

The naive reference point is a text-only formatting model applied to recognizer output. On Switchboard, Cho et al. report macro F1 of 0.597 for text-only prediction on ASPiRE recognizer output and 0.736 on gold transcripts. We reproduce this model with the same BERT encoder and data split before comparing anything.

The strongest existing method for the Switchboard setting is Cho et al.'s text-plus-prosody model, which reports macro F1 of 0.615 on recognizer output and 0.744 on gold transcripts. We reimplement it with the same recognizer hypotheses and give it the same tuning budget as our systems. For Spanish questions, the strongest method is the hybrid acoustic-lexical system of Zhu et al.. That paper reports only a relative gain of up to 4.4% question-mark F1, and no reported absolute question-mark F1 was found on Lacuna for the LDC Spanish Fisher corpus. We therefore rerun both its lexical-only and hybrid configurations ourselves.

The continuous-feature upper bound feeds WavLM features through a linear adapter into the same language model, following Wang et al., with an equal number of training steps.

Three ablations isolate the key ingredient. The first toggles duplicate removal while holding the codebook fixed, to separate the timing loss it causes from quantization loss. The second adds one more residual quantization layer at the same bitrate as the word-end side channel, to test whether targeted cues beat generic capacity. The third feeds the side channel with text only and no tokens, to test whether tokens add anything beyond Cho-style features.

Beating the baseline means two things. Tokens plus side channel must match or exceed 0.615 macro F1 on Switchboard recognizer output at a small fraction of the continuous bitrate. And cue retention must predict per-configuration F1 loss with clearly higher rank correlation than PESQ or UTMOS.

Why not just…?

Why not just let a large text model punctuate? Zhu et al. found PaLM2 reliable on only 28% of Spanish cases. They also show that statement-order questions are lexically unresolvable. Our text-only condition measures this ceiling directly on both corpora. If a strong text model matches the audio conditions on Switchboard, we will report that the Switchboard gains are small. That would be plausible, given Cho et al.'s +0.018. We would then shift weight to the Spanish question and incomplete-sentence classes, where text is provably insufficient.

Why not just use continuous features? They are the upper bound in our design, not a competitor we must beat on accuracy. Wang et al. report that discrete tokens cut bitrate by 99.9% and converge in 4–5 rather than 10–15 epochs, so pipelines built on tokens are not going away. The question is how much of the continuous–discrete gap a few numbers per word close. If the answer is "little," that is itself a useful negative result.

Why not just add codebook layers or a bigger vocabulary? This is a real threat, since AffectCodec finds that early layers carry the prosody. The bitrate-matched extra-layer ablation settles it. If one generic layer matches the targeted word-end side channel, we will conclude that the problem is capacity, not cue selection.

Why not just concatenate Cho-style prosody features and skip the audit? This may work as well as our repair, and we expect it often will. The contribution is therefore not that side channel. It is the diagnosis of whether a failure lies in the tokenizer or in a model ignoring the audio. That diagnosis tells a practitioner when concatenated features will help and when they will be ignored, as Koduru et al. and Pang et al. show can happen. The recoverable-but-unused measurement is what tests this.

Decisive Checks

Three results would confirm or refute the proposal, and all are pre-registered.

The first concerns duplicate removal on k-means tokens. If our hypothesis is right, it should cut listener and probe accuracy on incomplete and interruption boundaries far more than on questions. Coarse quantization should show the reverse pattern. If both operations degrade all boundary types alike, the claim that different components destroy different cues is wrong.

The second concerns prediction of downstream damage. Across all tokenizer configurations, word-end cue retention should correlate with downstream macro F1 loss more strongly than PESQ or UTMOS do. If holistic quality predicts as well, the perceptual lens adds nothing.

The third concerns the ordering of marks. Listener agreement on delexicalized original audio should rank the marks in the order question > sentence end > comma. If it does, that ordering would give a perceptual explanation for Zhu et al.'s finding that comma and period tokens in the recognizer's output raised its word error rate by up to 4.8%.