When people change topics in conversation, they rarely say so outright. Their voices do it for them: a long pause, a jump to a higher pitch, a louder onset. Often they also use a small discourse word such as "anyway", "so", "now" or "okay". In a transcript these words are treacherous, because the same "okay" that opens a new agenda item also serves as a mumbled acknowledgment in mid-topic. The two uses are told apart mostly by how the word sounds. This proposal asks a pretrained masked language model, meaning a model trained to fill in hidden words in text, to fill in the discourse word that would introduce the next utterance. The model sees the surrounding words plus a short description of the prosody, that is, the pitch, loudness and pause pattern at that point. A predicted word like "anyway" signals a major topic shift, "also" signals a subtopic step, and "yeah" signals continuation. Segmentation is thus reframed as predicting an unspoken connective, conditioned on how the speech sounds.
The bet is that this reuses two things the model already has: its pretrained knowledge of which connectives open new topics, and the graded strength of those connectives, which matches the graded size of pitch resets that phoneticians tie to discourse depth. If the bet holds, a few dozen labeled boundaries should be enough to train a segmenter that handles noisy speech recognition output and distinguishes major from minor boundaries. Feature-engineered classifiers do reasonably on the first problem and poorly on the second.
Core Claim
Fill-in-the-blank reformulation (here called cloze reformulation) has worked for written discourse. Prompt-based Connective Prediction inserts a masked slot between two sentences, has RoBERTa predict the missing conjunction, and maps that word to a discourse relation. It reports 67.79 Macro-F1 on top-level PDTB 2.0 senses against a prior best of 63.73, and raises the rare List relation from about 9% to nearly 30%. We transfer this mechanism from relations between written sentence pairs to boundaries in spoken, multi-party meetings.
The change that makes the transfer meaningful is the evidence placed in the prompt. The prompt carries discretized prosodic descriptors, for example "after a long pause, in a raised voice", taken from measured pause, pitch key and pitch span. Without them the model has only ambiguous words. The verbalizer, which is the mapping from predicted words to labels, is graded rather than binary. Connectives that open topics map to top-level boundaries. Additive or sequencing connectives map to sub-level boundaries. Backchannel and continuation words map to "no boundary".
The specific, testable claim has two parts. First, in a low-label regime, prosody-conditioned connective prediction will beat a pretrained encoder fine-tuned with a new linear output layer on the same words and the same prosodic features. Second, its advantage will be largest where prior systems are weakest: sub-level boundaries and utterances that begin with an ambiguous cue word.
Why this, and how it differs from what exists
The prosody literature motivates the graded verbalizer. The intonational Structuring of Discourse proposed that larger pitch-register resets mark deeper boundaries. Pitch range and pause duration as markers of discourse hierarchy confirmed perceptually that listeners use the relative size of pitch expansion and pause to separate major shifts from subtopics. Modeling the intonation of topic structure found that new topics are more than twice as likely to carry rising accents as continuations. All three describe a graded acoustic signal. Binary segmenters throw that gradation away.
The cue-word work motivates prosody-conditioning the slot itself rather than appending features at the end. Now let's Talk about Now and On the role of context and prosody in the interpretation of 'okay' show that discourse markers are disambiguated intonationally. Combining Multiple Knowledge Sources found that prosody filters the false alarms these words cause in lexical segmenters on the AMI meeting corpus.
The closest systems differ in clear ways. Hsueh and Moore, and the related AMI meeting facilitator, fuse prosody and words in a maximum-entropy classifier over hand-built features. That classifier has no pretrained language model and no few-shot evaluation, and it is reported to be substantially worse on sub-level topics. DialSTART and SumSeg use pretrained models for dialogue segmentation, but they are text-only and never pose the boundary as a fill-in task. Zhou et al. supply our core technique. They work on written sentence pairs and do no sequential segmentation.
Two results constrain the design. First, De Looze and Rauzy found that adding pitch key to pause gave no significant gain over pause alone. A naive "add pitch" story is therefore unsafe, and our pause-only baseline is a serious one. Second, Zhou et al. report large swings from trivial template changes, so every result must be averaged over several templates.
Research Program
For each candidate boundary $i$, meaning the start of a pause-delimited stretch of speech in the AMI convention, we build a prompt. The prompt contains the preceding context window and the upcoming utterance. Between them it places a prosodic descriptor string and a masked slot positioned where a sentence-initial connective would go.
The descriptor string is generated from four measurements, each binned into three or four ordinal levels. These are the preceding pause duration, the pitch key change, the pitch span change, and onset intensity. Pitch and intensity are normalized per speaker using the AMI speaker channels. This is cleaner than the story-level normalization used by Levow, which the research file notes breaks down in multi-party speech.
We take a boundary class $c \in \{\text{top}, \text{sub}, \text{none}\}$ with verbalizer word sets $V_c$ and score it as
$$s_c(i) = \log \sum_{v \in V_c} P(\text{[MASK]} = v \mid T_i),$$
where $T_i$ is the prompt for boundary $i$. Training uses a few labeled meetings and the decoupled label loss and label-conditioning regularizer of ADAPET. Under 32 labeled examples, ADAPET reaches a 77.3 SuperGLUE average without unlabeled data. Boundaries are then placed at local maxima of $s_{\text{top}}$ and $s_{\text{sub}}$ with a minimum-gap constraint.
We also build a second variant that feeds the raw continuous prosodic vector through a small projection into a few soft input positions instead of words. We contrast it with prototype-vector output classes in the style of ProtoVerb, which reported an 11.26-point absolute gain on 1-shot DBPedia with a frozen backbone. Comparing the two variants tells us whether describing prosody in words, which is how the model can actually use its pretrained connective knowledge, matters at all.
Baselines
The naive reference is pause thresholding. A top-level boundary is placed at every pause longer than a duration tuned on the few labeled meetings, and a second, shorter threshold marks sub-level boundaries. De Looze and Rauzy report pause-alone F-measures of 0.321 to 0.766 across four corpora. On AMI we will compute this baseline ourselves, since no reported result exists on Lacuna for pause-only segmentation of AMI. The proposal fails if it does not clearly beat this threshold on sub-level boundaries.
The strongest existing reported system is the Hsueh and Moore maximum-entropy model with all feature streams. It reaches $P_k = 0.30$ on AMI top-level topics with reference transcripts and 0.31 with 39.1%-WER recognition output. $P_k$ is the probability that two points a fixed distance apart are wrongly judged to be in the same or different segments; lower is better. Their lexical-only LCSEG baseline reaches 0.40. We re-implement the model on its prosodic, conversational and lexical streams but omit video, so the comparison is not inflated by modality. We train it on the identical few-shot splits and on full training data. Beating it means lower $P_k$ and WindowDiff at matched label counts. With full data, a top-level $P_k$ at or below 0.30 counts as parity, and any sub-level gain counts as a contribution, since no sub-level number is reported.
The strongest modern competitor is RoBERTa-large fine-tuned with a new linear output layer over the same window, with the same continuous prosodic vector concatenated to the pooled representation. It uses the same label budget, the same number of hyperparameter trials, and the same number of random seeds. For text-only modern segmentation we run DialSTART unsupervised on AMI transcripts. DialSTART reports $P_k$ of 17.86% on DialSeg711, but no result is reported on Lacuna for it on AMI.
Three ablations isolate the key ingredients. The first removes the prosodic descriptor from the cloze prompt. The second replaces the graded connective verbalizer with the binary words "shift" and "continue". The third replaces the verbalized prosody with soft projected vectors.
Why not just…?
Why not just fine-tune an encoder with a classification head on words plus prosodic features? This is the real threat. With enough labels it may well match the cloze model, because the objective-gap arguments in ADAPET and Prompt-and-Align concern scarce data. We therefore frame the claim as a learning curve rather than a single number. We train both models at 2, 5, 10, 20 meetings and full data, with identical tuning budgets. If the curves cross before about 10 meetings and the head-based model wins thereafter, we will report that the benefit is purely sample efficiency.
Why not just threshold pauses, since pitch added nothing in prior work? De Looze and Rauzy's null result for pitch is direct evidence that this baseline could win on top-level boundaries. Levow's tree still reached 89.4% balanced accuracy on Mandarin news without silence features, which shows pitch and intensity carry signal when pauses are uninformative. Our sub-level and cue-word slices test exactly the cases where pause length is ambiguous. If pause thresholding ties the full model there too, the prosodic descriptors are not contributing.
Why not just run a strong text-only segmenter on the transcript? Text-only systems cannot see why an "okay" was said. Ohtsuki et al. (Topic structure extraction for meeting indexing) show text segmentation collapsing from 66.7% to 16.7% exact accuracy under poor recognition. The no-prosody ablation together with the recognition-output condition settles whether our model inherits that fragility.
Proposed Experiments
The main benchmark is the AMI Meeting Corpus, at both top and sub levels. We evaluate on reference transcripts and on the 39.1%-WER recognition output, using $P_k$ and WindowDiff. Results are averaged over five templates and five seeds, and we report the variance across templates.
Three checks decide the outcome. The first is the few-shot learning curve against the head-based model. The second is a cue-word slice: we compute false-alarm rate on utterances beginning with "okay", "now", "so" or "right", where the claim predicts the largest gap between the cloze model with and without prosody. The third is a depth check, testing whether $s_{\text{top}} - s_{\text{sub}}$ correlates with measured pitch-reset magnitude on held-out boundaries, as the hierarchy theory predicts.
As a secondary transfer test, we train on AMI and evaluate with a multilingual masked model on TDT-3 Mandarin broadcast news. The verbalizer is translated, and prosodic binning is unchanged. There we compare against Levow's reported 95.8% balanced accuracy only in her balanced-sample protocol, because that protocol is not comparable to sequential $P_k$.
References
- Prompt-based Connective Prediction Method for Fine-grained Implicit Discourse Relation Recognition
- The intonational Structuring of Discourse
- Pitch range and pause duration as markers of discourse hierarchy: perception experiments
- Modeling the intonation of topic structure: two approaches
- Now let's Talk about Now; Identifying Cue Phrases Intonationally
- On the role of context and prosody in the interpretation of 'okay'
- Combining Multiple Knowledge Sources for Dialogue Segmentation in Multimedia Archives
- Automatic Segmentation and Summarization of Meeting Speech
- Unsupervised Dialogue Topic Segmentation with Topic-aware Contrastive Learning
- Leveraging Summarization for Unsupervised Dialogue Topic Segmentation
- Automatic detection and prediction of topic changes through automatic detection of register variations and pause duration
- Prosody-based Topic Segmentation for Mandarin Broadcast News
- Improving and Simplifying Pattern Exploiting Training
- Prototypical Verbalizer for Prompt-based Few-shot Tuning
- Prompt-and-Align: Prompt-Based Social Alignment for Few-Shot Fake News Detection
- Topic structure extraction for meeting indexing