A dialogue-act tagger labels each turn in a conversation as a statement, question, backchannel ("uh-huh"), and so on. In practice it runs on top of automatic speech recognition (ASR), the system that turns audio into text. Moving such a tagger from one speech platform to another, say from telephone calls to multi-party meetings, changes three things at once. The recognizer makes more or different errors. People take turns differently. The microphone channel distorts pitch and loudness. Current systems mix these three effects into one accuracy drop, so nobody can say which to fix. We propose to separate them with a model whose parts already line up with them: a conditional random field (CRF). A CRF scores each turn's label from local evidence and also scores which label tends to follow which. It therefore keeps "what was said and how" apart from "what usually comes next."
The core idea is a CRF tagger with three swappable pieces. The first is a lexical scorer whose influence is turned down when the recognizer is unsure of its transcript. The second is a prosodic scorer that reads speaker-normalized pitch, energy, pause and duration features. The third is a table of label-to-label transition scores that we re-estimate on the new platform from unlabeled audio alone. Swapping each piece between its source-platform and target-platform version gives an error budget for cross-platform transfer. Fixing the two pieces we expect to carry most of the damage, lexical over-trust under ASR noise and turn-taking norms baked into the transitions, should recover much of the gap without any target labels.
Core Claim
The claim is that cross-platform loss in spoken dialogue-act tagging comes mostly from two separable sources, each correctable without target labels. The first is lexical evidence that becomes unreliable as upstream recognition errors grow. The second is label-transition statistics that encode one platform's turn-taking habits. A CRF with an ASR-confidence-gated lexical term, a prosodic term, and transition scores re-estimated on target data will therefore transfer better than an otherwise identical tagger that fuses the same features in one encoder. The strongest version of this test also gives that competitor ASR-noise training and confidence features, and it must still lose.
Concretely, for a conversation of turns $t=1..T$ with labels $y_t$, the tagger scores a label sequence as
$$s(\mathbf{y}) = \sum_t \Big[ g(c_t)\, E^{\text{lex}}_t(y_t) + E^{\text{pros}}_t(y_t) + A^{(p)}_{y_{t-1},y_t} \Big].$$
Here $E^{\text{lex}}_t$ is the lexical score from the ASR transcript of turn $t$, and $E^{\text{pros}}_t$ is the prosodic score. $c_t$ is the recognizer's average word confidence for that turn. $g$ is a learned non-decreasing function in $[0,1]$. $A^{(p)}$ is the transition table for platform $p$. Training and exact decoding use the standard forward-backward and Viterbi procedures. Because conversations have only a handful of coarse labels, the cost that grows with the square of the label-set size is negligible here.
Prior Work and What It Leaves Open
Three strands motivate the design without combining it. Kumar et al.'s hierarchical BiLSTM-CRF encodes words into turn vectors, encodes turns into conversational context, and decodes acts with a conversation-level CRF. It reports 79.2% accuracy on the Switchboard Dialogue Act corpus (SwDA) and 90.9% on the ICSI Meeting Recorder corpus (MRDA), and replacing the CRF with independent softmax costs roughly 6%. It is text-only, uses gold transcripts, and never moves between corpora. It supplies our backbone.
Shriberg et al. showed on Switchboard, where recognition error exceeded 30%, that prosody is a safety net. Their 7-way accuracy was 41.15% from prosody alone, 67.6% from words, and 69.9% combined. Duration separated backchannels from agreements. They used decision trees over isolated, hand-segmented turns, with no sequence model and one platform. They supply our prosodic feature set and the hypothesis that prosody degrades more gracefully than words.
Huang and Chen's lattice language model attacks error propagation by feeding recognition lattices, weighted by recognizer confidence, into the downstream encoder. It reports 3% to 42% relative error reduction on SwDA, MRDA, ATIS and SNIPS. It uses no prosody, no CRF and no platform shift. We borrow its premise that recognizer confidence should modulate downstream trust, but apply it to how much the decision relies on words rather than to the encoder input.
Two further results constrain the design. On the platform side, CATCH separates platform-dependent from platform-invariant text features for hate speech across Gab, Reddit, Twitter and YouTube, but needs target labels and ignores speech. Miller et al. show that small per-component losses under domain shift compound through a pipeline. That is exactly why we expect a worse recognizer on the target platform to hurt the tagger more than either effect alone. On the encoder side, Li et al. and Mischler et al. find that deep layers of speech encoders discard pitch and emotional dynamics. Our learned-audio baseline therefore reads intermediate layers, not the top.
Research Program
The two platforms are telephone dialogue (Switchboard, with SwDA labels) and multi-party meetings (ICSI, with MRDA labels). Both have audio. They differ in channel, number of speakers and turn-taking, for example MRDA's floorgrabbers and disruptions. Their label sets differ (42 versus 5 classes), so we will define and release a shared coarse mapping covering statement, question, backchannel, and an "other" class absorbing floorgrabbers, disruptions and remaining acts. We will report results per class so the mapping choice is visible. One off-the-shelf recognizer, unadapted, will transcribe both corpora, and we report word error rate per corpus. Its N-best or confusion-network posteriors give $c_t$.
The prosodic scorer uses Shriberg-style features: F0 mean, range and slope, energy, voiced duration, preceding and internal pause length, and speaking rate. Each is z-normalized per speaker within a conversation, which is meant to absorb channel differences. We will also try inserting pause tokens into the lexical stream, following Yuan.
Transfer to the target platform needs no labels. We fix the emission scorers and estimate $A^{(p)}$ on unlabeled target conversations by maximizing marginal likelihood with forward-backward posterior counts. We regularize toward the source table to prevent drift.
The analysis contribution is the error budget. We run the full grid of source-versus-target versions of each piece: gold-versus-ASR transcripts for the lexical term, source-versus-oracle-target transitions (estimated from target labels purely for diagnosis), and prosody on versus off. From this grid we attribute the in-domain-to-cross-domain accuracy gap to lexical/ASR, transition, and channel effects, in both transfer directions.
Baselines
The naive reference point is the source-trained text-only hierarchical BiLSTM-CRF of Kumar et al., applied unchanged to target-platform ASR transcripts. Its in-domain gold-transcript numbers (79.2% SwDA, 90.9% MRDA) are the published reference. I found no reported result on Lacuna for cross-platform SwDA-to-MRDA or MRDA-to-SwDA transfer, nor for either corpus under ASR transcripts with this model. We will re-run it on the coarse label set so that all numbers are comparable.
The strongest competitor is a matched early-fusion model. It uses the same hierarchical encoder and CRF, but the prosodic features and the confidence $c_t$ are concatenated into each turn vector, and it is trained on ASR transcripts of the source platform. This is what a careful practitioner would build. It has access to every input our model has and differs only in letting one encoder learn the weighting. It gets the same features, data, number of hyperparameter trials, and seeds, and it may also use its own unlabeled-target transition re-estimation, so that the gate is tested in isolation.
We add a learned-audio baseline that replaces handcrafted prosody with intermediate-layer wav2vec 2.0 features pooled per turn under the same CRF, again following the layer findings of Li et al.. We also add the lattice encoder of Huang and Chen as the strongest pure error-propagation fix.
Ablations isolate each ingredient: gate fixed to 1, prosody removed, and transitions set to source, uniform, re-estimated, or oracle. An upper bound fine-tunes on 5% of labeled target conversations. Beating the strongest competitor means higher cross-platform macro-F1 in both directions across five seeds, with the gain concentrated where target recognition error is higher.
Why not just…?
Why not just concatenate prosody and confidence into the encoder? This is the real threat, and it is the strongest baseline above. Our argument is that a weighting learned on source data is calibrated to source recognition error. When the target recognizer is worse, a fused encoder keeps trusting words at the source rate, whereas an explicit gate reads the new confidence values directly. The test injects controlled additional recognition error on the target side by narrowing the decoding beam and adding noise. We predict the fused model's deficit grows with error rate. If it does not, and the fused model matches ours, we will report that the gate is unnecessary. The contribution then shrinks to the error budget and transition re-estimation.
Why not just use an end-to-end audio language model? Hear2Act finds that audio LLMs classify prosodic states at about 85% accuracy yet act on them barely better than with transcripts (15.3% versus 14.6%). Adding an explicit intermediate state raised this to 39.6%. That argues for exposed intermediate structure, though it is a different task with synthetic speech. Sperber et al. note that paired speech-to-label data is scarce. We do not dismiss this option. The learned-audio CRF baseline measures how far learned features take us, and an audio LLM run zero-shot on the coarse labels is reported as a reference.
Why not just drop the CRF, since contextual encoders already see neighbors? Kumar et al. report about a 6% loss without it in-domain. Cross-platform, however, a wrong transition table could hurt more than none, which is precisely our claim. The source-versus-uniform-versus-re-estimated comparison settles this directly.
Why not just train on noisy ASR transcripts and skip everything else? Noise-matched training is built into the strongest baseline, so any gain we claim is on top of it.
Decisive Checks
Four results would establish or sink the proposal.
First, the error budget must account for most of the transfer gap, and the lexical and transition components must dominate it. If the channel component dominates, the premise is wrong.
Second, prosody-only tagging must show a smaller relative cross-platform drop than words-only tagging on ASR transcripts. Shriberg et al. imply this holds within Switchboard, but it has never been tested across platforms, so it could fail.
Third, re-estimated transition tables must move measurably toward the oracle target tables in KL divergence, and must improve accuracy over source transitions. Gains should be largest for backchannels and MRDA-specific acts.
Fourth, the gated model's advantage over the fused competitor must increase monotonically with target word error rate under controlled noise injection. A flat curve would falsify the error-propagation part of the claim.
References
- Dialogue Act Sequence Labeling Using Hierarchical Encoder With CRF
- Can Prosody Aid the Automatic Classification of Dialog Acts in Conversational Speech?
- Learning Spoken Language Representations with Neural Lattice Language Modeling
- Causality Guided Disentanglement for Cross-Platform Hate Speech Detection
- Domain adaptation in practice: Lessons from a real-world information extraction pipeline
- Exploration of a Self-Supervised Speech Model: A Study on Emotional Corpora
- Parallel hierarchical encoding of linguistic representations in the human auditory cortex and recurrent automatic speech recognition systems
- Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does
- Speech Translation and the End-to-End Promise: Taking Stock of Where We Are