Prosodic Cohesion: Scoring Where Synthetic Speech Breaks Its Phrases

Reference-free speech quality predictors take a clip of synthesized, converted or enhanced speech and output a human-like quality score without a clean recording to compare against. They are now very good at hearing noise, buzz and spectral distortion. They have no explicit notion of where in the sentence a pause or hesitation lands. Listeners do. "The old man | walked home" sounds natural. "The old | man walked home" sounds broken, even though both clips can contain the same amount of silence and the same acoustic cleanliness.

We propose to import an old idea from machine translation to measure this. Syntactic cohesion is the rule that once you start rendering a grammatical unit (a subtree of the sentence's dependency parse, meaning a head word plus everything that depends on it), you should finish it before jumping elsewhere. We turn that rule into a reference-free "prosodic cohesion" score. The score counts how much break strength (pauses, pre-boundary lengthening) the audio places inside tight subtrees, and how much it withholds at clause boundaries where natural speakers do break. The claim is that this score captures a failure that current predictors weight weakly, and that adding it helps most on long, multi-clause utterances and on out-of-domain systems.

Core Claim

Phrase-internal breaks and missing clause-boundary breaks are a distinct perceptual defect in text-to-speech (TTS), voice conversion and packet-loss-affected speech. We can measure this defect without a reference, by aligning the audio to its words, parsing the words, and applying a soft, depth-aware cohesion penalty borrowed from statistical translation decoding. The novelty is not that prosody carries syntax, which is already established. The novelty is using syntax-prosody agreement as a quality signal, with the soft-penalty and depth-binning machinery that made cohesion constraints work in translation.

Where This Sits

Three strands of prior work supply the method.

Cohesive Phrase-Based Decoding for Statistical Machine Translation penalized translation hypotheses that split a source subtree. It found that hard constraints were too restrictive and soft penalties were needed. With soft penalties it gained 0.5 to 1.1 BLEU on the hardest 20% of sentences, and humans preferred cohesive output 47% to 8%.

Cohesive Constraints in A Beam Search Phrase-based Decoder refined this into counts of how many words remain uncovered in an interrupted subtree.

Soft Dependency Constraints for Reordering in Hierarchical Phrase-Based Translation showed that separating dependencies near the root from deep, local ones ("depth binning") matters.

We reuse all three ingredients. In our setting a "break inside a subtree" replaces "a translation jump out of a subtree," and depth decides how bad an interruption is.

Using Prosody to Predict Syntactic Structure motivates the proposal. It shows that word durations and pauses carry roughly 2 to 3.5 nats of syntactic information and mostly predict phrase boundaries rather than labels. That is exactly the signal a break-placement score needs. However, it only measures information in natural speech and never scores degraded or synthetic speech, which is our step.

On the quality side, UTMOS, APG-MOS and NISQA predict opinion scores from self-supervised embeddings, cochlea-like filterbanks or spectrograms. None of them conditions on the sentence's structure. UTMOS's own ablation found that phoneme inputs added little in-domain. This constrains our design: feeding in text is not enough, and the structure must be used in a targeted way.

The closest bridging work, ARECHO, models dependencies between quality metrics as an autoregressive chain. We deliberately do not pursue the obvious extension of turning that chain into a tree of metrics. ARECHO's random-permutation training already lets any metric condition on any other, so a tree would mostly restrict it. Our dependency structure lives inside the utterance, not among the metric outputs.

Research Program

The pipeline has four stages: transcript, dependency parse, forced alignment, then the cohesion score.

We first obtain a transcript. We use the synthesis input text when a corpus distributes it, and otherwise an automatic transcript. We parse the transcript into dependencies. We then align words to audio with the Montreal Forced Aligner, following Using Prosody to Predict Syntactic Structure.

At each word juncture $j$ we compute a break strength $b_j$ from the audio: silence duration plus speaker-normalized lengthening of the preceding word. We also compute a syntactic depth $d_j$, defined as the depth of the smallest subtree that contains both neighboring words. A small $d_j$ marks a clause-level seam; a large $d_j$ means the juncture sits inside a tight unit such as determiner and noun.

Rather than hand-set penalties, we learn from natural read and conversational speech (LibriTTS and CANDOR) the expected break strength $\mu(d, r)$ for each depth bin $d$ and dependency relation $r$. We then score an utterance by its two one-sided deviations:

$$ S_{\text{int}} = \sum_j w_{d_j}\,\max(0,\ b_j - \mu(d_j, r_j)), \qquad S_{\text{omit}} = \sum_j w_{d_j}\,\max(0,\ \mu(d_j, r_j) - b_j). $$

Here $S_{\text{int}}$ is interruption mass (spurious breaks inside units), $S_{\text{omit}}$ is omission mass (run-on speech across clause seams), and $w_{d}$ are per-depth-bin weights fit on a held-out development split. This is the soft, depth-binned analogue of the interruption counts in Cohesive Constraints in A Beam Search Phrase-based Decoder. It stays soft because, as Cohesive Phrase-Based Decoding for Statistical Machine Translation found, parser errors and legitimate variation make hard barriers destructive.

The two scores, together with per-bin breakdowns, enter a quality predictor as a small side input. We add them in two ways. In the first, they join a weak learner in a UTMOS-style stack. In the second, they are concatenated before the pooling layer of an APG-MOS-sized model. The two routes let us check whether the signal survives in both a large ensemble and a compact model.

We also study robustness. We compare gold text against automatic transcripts, and clean parses against parses from a deliberately degraded parser. The aim is to show the score degrades gracefully. That matters because the reason for using prosody at all is that it partly corrects noisy lexical input, as Using Prosody to Predict Syntactic Structure observed.

Baselines

Naive reference points. The first predicts the training-set mean opinion score. The second is a syntax-free pause regressor that uses total pause duration and pause count from the same aligner. Any gain from our score must exceed what raw pause statistics give.

Strongest existing methods. The strongest large competitor is UTMOS, which reports utterance-level MSE 0.165 and SRCC 0.897 on the VoiceMOS 2022 main track. The VoiceMOS Challenge 2022 reports top system-level SRCC around 0.939. The strongest compact competitor is APG-MOS, with SRCC 0.936 on BVCC and 0.921 on SOMOS. For NISQA's discontinuity dimension, the reference is NISQA itself, which reports RMSEs of 0.16 to 0.56 across dimensions; the file does not give the discontinuity-specific value.

We will retrain each baseline with and without our features on identical splits. Each gets the same tuning budget, measured in hyperparameter trials. We use three seeds and report bootstrap confidence intervals.

What counts as winning. Clearly matching published headline numbers is not the bar. Beating a baseline means a statistically significant gain on the long-utterance subset (three or more clauses) and on held-out synthesis systems, with no loss on the full set.

Ablations that isolate the key ingredient. The first replaces the real parse with a random tree of the same length. The second replaces depth bins with a single bin. The third uses a hard violation count, following the brittle design that Cohesive Phrase-Based Decoding for Statistical Machine Translation warns against. If the random-tree variant matches the real one, the syntax is doing nothing and the claim fails.

Why not just…?

Why not just trust self-supervised predictors, which already hear prosody implicitly? This is the real threat. UTMOS-style models may already encode break placement. The evidence against is indirect. UTMOS itself reports that text-side inputs added little, which suggests it uses structure weakly rather than well. The VoiceMOS Challenge 2022 documents out-of-domain degradation. Our controlled perturbation test, described below, settles this directly. If UTMOS already separates phrase-internal from phrase-boundary pauses as sharply as listeners do, we will report that the gap does not exist.

Why not just count pauses without a parser? Pause statistics catch hesitant or choppy speech but cannot tell a well-placed pause from a misplaced one of equal length. The syntax-free pause regressor and the random-tree ablation are exactly this alternative. The perturbation set holds total pause duration fixed so that only placement differs.

Why not just score prosody by its likelihood under a model of natural speech? The diffusion-likelihood approach of Non-intrusive Speech Quality Assessment with Diffusion Models Trained on Clean Speech and the quantization-error approach of Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech both score deviation from natural speech. A text-conditioned version could learn break placement without our explicit depth bins, and this is plausible. We therefore include a learned competitor that scores synthetic speech by that conditional cross-entropy. It is a prosody-given-syntax model in the style of Using Prosody to Predict Syntactic Structure, trained on LibriTTS and CANDOR. If it matches our score, we will adopt it and treat the depth-binned score as its interpretable diagnostic rather than as the better predictor.

Why not just extend ARECHO's metric chain into a dependency tree? ARECHO already conditions metrics on one another, cutting MSE from 96.99 to 23.26 against a parallel baseline. A tree over a handful of metric outputs adds constraint without adding information about the signal. Our proposal adds a new input signal instead, and it can be combined with ARECHO's decoder as an extra token.

Proposed Experiments

Decisive first check: controlled perturbation and listening test. We take natural LibriTTS sentences and fluent TTS renditions. We insert or lengthen pauses either inside deep subtrees or at clause seams, holding total added silence fixed. We also remove clause-seam breaks by time-compressing the pauses at those seams. A crowdsourced listening test rates each version. Three outcomes are possible. If listeners do not penalize misplacement, we stop. If listeners do and UTMOS and APG-MOS do not, the gap is confirmed. In either case the cohesion score should track the human differences.

Main benchmarks. We then evaluate on BVCC (main track and held-out systems), SOMOS, and the discontinuity and overall dimensions of the four NISQA test sets. For NISQA we use automatic transcripts, since packet loss that lands inside a noun phrase should be scored worse than loss between clauses. Metrics are utterance- and system-level SRCC and MSE, reported overall, stratified by clause count, and for unseen systems.

Residual check. We measure the partial correlation of $S_{\text{int}}$ and $S_{\text{omit}}$ with human scores after regressing out UTMOS predictions. If this partial correlation is significant, the score carries information the strongest predictor lacks. If it is not, the score is at best a readable explanation of what the predictor already knows.