When a patient says "I think I had chest pain last week" or a doctor says "could be a reflux thing," the condition is neither present nor absent. It is possible. Clinical extraction systems have to label every mentioned condition this way. This labeling task, called assertion detection, decides whether a condition is present, absent, hypothetical, or uncertain. The uncertain "possible" class is rare and is where text models do worst. More clinical documentation now starts as recorded conversation that is transcribed and then turned into a note. In speech, uncertainty is often marked by how something is said: a pause before the symptom word, a lengthened syllable, a rise in pitch at the end of a phrase. These are suprasegmental or prosodic cues, meaning patterns of pitch, timing, and loudness that span syllables and words rather than individual sounds. Converting speech to a transcript throws them away.
We propose to test one specific claim. Prosodic features measured on the words of an entity mention, relative to the same speaker's baseline, add evidence for the rare "possible" assertion class. That evidence is not recoverable from the transcript and not recoverable from pause timing alone. It should help most where text models fail, namely when the lexical hedge word is missing, far from the entity, or ambiguous. The work builds a spoken assertion corpus from real-audio clinical dialogues. It then runs a ladder of models that add prosodic information in controlled steps, and asks whether any gain survives against a strong, rebalanced text model and an end-to-end audio model.
Core Claim
Text-only assertion work motivates the gap. Assertion Detection in Clinical Notes: Medical Language Models to the Rescue? evaluates assertion detection on discharge summaries and other notes. It finds that domain-pretrained BERT models gain up to about 15 points of F1 on the "possible" class, yet 20% of the remaining errors come from long-range separation between the hedge cue and the entity, and 45% from annotator inconsistency. Joint Entity Extraction and Assertion Detection for Clinical Text conditions assertion predictions on predicted entities. It reaches 0.905 negation F1 on i2b2 2010 but leaves speculative assertions largely unresolved. Both papers work only on written text.
Our claim is that in speech, the uncertainty signal is often physically on the entity words, as hesitation and lengthening around the mention. A cue located on the mention itself sidesteps the long-range problem.
The claim is also narrower than "audio helps clinical notes." Speech-to-SOAP: End-to-End Summarization of Medical Dialogues: KIT@BeTraC 2026 already maps raw audio to SOAP notes (Subjective, Objective, Assessment, Plan) with an audio-language model, on the argument that transcripts discard paralinguistic cues. It never isolates prosody, never evaluates assertion status, and trains largely on synthetic text-to-speech audio. That last point matters a great deal: synthesized speech carries no genuine patient hesitation. We differ from that work in three ways. We extract explicit word-level prosodic measurements, we evaluate a rare-class assertion target, and we restrict all training and evaluation of prosodic effects to real recorded speech.
Research Program
Corpus. We will assemble the real-audio portions of the dialogue collections unified in the KIT@BeTraC 2026 data (Speech-to-SOAP cites PriMock57, ACI-Bench, and MTS-Dialog among them). Every synthetic Kokoro-rendered recording will be excluded. We will annotate condition mentions with i2b2 2010-style assertion labels.
Two annotators will label each mention twice: once from the transcript alone, and once while listening to the audio. The second pass serves two purposes. It measures directly whether humans change their assertion judgment when they hear the prosody. It also gives a label set less affected by the transcript-only ambiguity that drives much of the error reported in the Medical LMs study. The share of mentions whose label flips between the two passes is the first result. If almost none flip, the hypothesis is weak before any model is trained.
Features. Following the point in Keikichi Hirose et al. that prosody is relative, we align audio to the transcript and compute features for each entity span and its surrounding words. The features are pitch slope and range, syllable duration, energy, preceding pause length, and phrase-final pitch movement. Each is normalized against the same speaker's statistics within the encounter, which addresses the speaker and dialect confounding described by Helen Meng et al..
Pitch tracking is unreliable on creaky or reverberant speech, a failure noted in Using mutual information to identify regions of analysis for prosodic analysis. We will therefore keep a voicing-confidence mask and report results on low-confidence spans separately. That same paper also supplies a method we reuse: choosing the analysis window around the target by mutual information rather than fixing it at the word.
Models. The text branch is a clinical BERT assertion classifier first trained on i2b2 2010 and the MIMIC-III assertion corpus, then fine-tuned on transcripts. Pretraining on large written corpora addresses the rarity of the "possible" class: the text model learns lexical hedging from abundant data, and the small spoken corpus only needs to teach a lightweight prosody adapter a residual correction.
The prosody branch comes in two strengths. The first is the hand-computed relative feature vector above. The second is a learned encoder: the frame-to-phone-to-word stages of SpeechFormer++ (paper, code), pooled over the entity span. A gate fuses the prosody branch into the text representation of the entity. Rebalancing (class-weighted loss and minority resampling) is applied identically to every arm, so that any prosody effect is not confused with a rebalancing effect.
Baselines
The naive reference point is the majority-class predictor, plus NegEx-style rules applied to ASR transcripts. On written i2b2 text, rules reach 0.845 negation F1 against 0.905 for joint modeling (Joint Entity Extraction and Assertion Detection for Clinical Text). The rules show how much of the problem is solved by keyword cues.
The strongest text competitor is the medical-pretrained BERT assertion model from Assertion Detection in Clinical Notes. We will give it the same i2b2/MIMIC pretraining, the same spoken fine-tuning data, the same rebalancing options, and the same hyperparameter search budget as the fused model. We will run it on both manual and ASR transcripts. That paper reports gains of up to about 15 points of F1 on "possible" over its baselines on written notes. There is no reported result on Lacuna for assertion detection on spoken clinical dialogue, so its number on our corpus must be measured. Beating this model means a statistically reliable gain in "possible"-class F1, using bootstrap over encounters, with "present" and "absent" no worse.
The strongest audio competitor is an end-to-end audio-language model in the style of Speech-to-SOAP: Qwen2.5-Omni-3B with LoRA. We will fine-tune it on the same real-audio training split to output assertion labels for given mentions, with the same rebalancing applied. The only related figure in the file is its Concept-F1 of 0.4986 on SOAP generation (dev set), which measures a different task. This model tests whether implicit audio modeling already captures what our explicit features capture.
The key ablation replaces all prosody with timing tokens. Pause markers and word-duration bins from ASR timestamps are inserted into the transcript fed to the text model. This isolates whether pitch and relative lengthening add anything beyond "there was a pause."
Why not just…?
Why not just transcribe with ASR and run the best clinical text model? Hedging is often lexical ("maybe," "I think"), so this is a serious threat. We treat it as the main baseline rather than a straw man. The evidence that it falls short comes only from written notes: errors from long-range cue separation and annotator inconsistency in Assertion Detection in Clinical Notes. Whether it falls short on speech is exactly what the experiments measure. If the fused model gains nothing on mentions without a nearby lexical hedge, the text model wins and we will report that.
Why not just add pause tokens to the transcript? This is the cheapest prosody, and it may capture most of the signal, because hesitation is the most obvious uncertainty cue. It is built in as the key ablation arm. If timing tokens match the full prosody model, the contribution narrows to "timing is enough." That would still be a useful finding for deployed systems. We would then drop the pitch and learned-encoder components rather than defend them.
Why not just fine-tune an end-to-end audio model with class weighting? Speech-to-SOAP shows these models can learn from audio. However, its own results show degradation on long encounters and no benefit from chain-of-thought prompting, and it was trained largely on synthetic audio that lacks real prosody (Speech-to-SOAP). Our head-to-head run on identical real-audio data settles whether an explicit span-level signal does better on the rare class. A training corpus of a few thousand mentions may favor the explicit features, but that is a prediction, not a result.
Why not just oversample the rare class harder? Oversampling can lead models to memorize the few minority examples (Adrian Tousignant et al.). Rebalancing also adds no new evidence about the minority class. Every arm receives the same rebalancing, so the comparison measures what prosody adds on top of it.
Proposed Experiments
The decisive check is stratified. We split test mentions of the "possible" class by whether a lexical hedge occurs within a few tokens of the entity, and compare fused-versus-text F1 within each stratum. The hypothesis predicts the gain concentrates in the no-nearby-hedge stratum, which corresponds to the long-range failure in the written-notes analysis. A uniform small gain would suggest the model is mostly learning speaker or recording style rather than uncertainty marking.
Three secondary checks follow.
The first is the human listen-versus-read label flip rate. We then test whether model gains concentrate on the flipped mentions.
The second is a shortcut test inspired by DAIC-WOZ: On the Validity of Using the Therapist's prompts, which showed text models exploiting interviewer prompts. We compute prosody only from the turn of the speaker who uttered the mention, then separately from the other speaker's turns. Gains that come from the doctor's question prosody when the patient asserts the condition would signal a shortcut rather than the mechanism.
The third is robustness. We report results on ASR rather than manual transcripts, and separately on low pitch-confidence spans.
Metrics are per-class F1 with "possible" as the primary target, macro-F1, and "possible"-class recall at the precision of the text baseline. All are reported with encounter-level bootstrap intervals. To stress the rebalancing claim, we will also vary the "possible" share of the fine-tuning data by subsampling.
References
- Assertion Detection in Clinical Notes: Medical Language Models to the Rescue?
- Joint Entity Extraction and Assertion Detection for Clinical Text
- Speech-to-SOAP: End-to-End Summarization of Medical Dialogues: KIT@BeTraC 2026
- Use of prosodic features for speech recognition
- Studying L2 suprasegmental features in asian Englishes: a position paper
- Using mutual information to identify regions of analysis for prosodic analysis
- SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing
- DAIC-WOZ: On the Validity of Using the Therapist’s prompts in Automatic Depression Detection from Clinical Interviews