When a speech editor replaces a word in a recording, or a streaming synthesizer emits audio chunk by chunk, the weak point is the seam. This is the few tens of milliseconds where generated audio meets real audio. A click, a pitch jump, or a sudden change in room tone there can make listeners reject an edit that is otherwise perfect. The metrics the field reports cannot see this. Mel-cepstral distortion (a frame-by-frame spectral distance to a reference), perceptual quality scores like PESQ, and word error rate all average over the whole sentence, so a 10 ms defect disappears into the mean. We propose to build a small learned detector that looks only at a short window around each seam and outputs the probability that a listener would hear a join there. The detector judges each seam against how real speech behaves across the same kind of phonetic transition. We will use it to measure editing systems, and as a cheap critic that picks the cleanest of several candidate edits.
Expressive speech is what makes this hard, and it is why naive seam detectors fail. Natural speech is full of abrupt events. Stop consonants produce broadband bursts, and emphatic stress produces large pitch excursions. These look exactly like splice artifacts to an edge filter or a pitch-jump threshold. A seam detector is only useful if it separates "abrupt but natural" from "abrupt because two pieces were glued together." The detector should also stay quiet when an edit deliberately changes emotion, the case where global prosody-consistency losses misfire. Our claim is that this separation can be learned from automatically generated splices of natural expressive speech, then calibrated with a modest set of human seam-detection judgments.
Core Claim
A seam-local detector has three properties. It sees a short window centred on the boundary. It is trained to tell real phone transitions from spliced ones within the same phonetic class. It is calibrated on human "did you hear a join?" judgments. Such a detector will predict human fluency ratings of speech edits much better than utterance-level objective metrics. Used to rerank sampled edits, it will recover much of the quality gain that slow audio-language-model critics provide, at a small fraction of the cost.
The novelty is not a new smoothing loss. Several papers already add boundary losses. The novelty is a calibrated, context-conditioned measurement of the boundary itself, which no paper in our material provides.
Prior Work and Positioning
Three editing systems motivate the gap.
- FluentEditor2 and FluentEditor+. These train a diffusion editor with frame-, phoneme- and word-level smoothness costs plus a contrastive term that pulls edited prosody toward the original utterance. They evaluate with utterance-wide MCD/PESQ/STOI and listener fluency scores. Their contrastive term also penalizes intended local emotional shifts.
- VoiceCraft. This infills codec tokens with a large language model. It reports that low word error rate can coexist with unnatural speech, and that outputs contain sporadic silences and scratching sounds, which users fix by resampling seeds.
- FastStitch. This blends mel-spectrograms with a convolutional or attention network, but is limited by duration-predictor timing errors.
All three treat the seam as something to suppress during training. None measures it afterwards in a way that agrees with listeners. We build the missing measurement and do not compete on the generator.
Two works supply method.
- A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences. This trains a convolutional network on roughly 55,000 human same/different judgments, collected by active learning, to get a perceptual distance robust to misaligned signals. We reuse its recipe for gathering judgments near the detection threshold. We apply it to localized seams rather than global degradations such as noise and reverberation.
- Mesostructures. This shows that joint time-frequency scattering (a two-dimensional wavelet analysis of the spectrogram that captures how energy patterns change over time) gives distances that stay smooth under time shifts, whereas multi-scale spectrogram distances become jagged. That matters here because seam location estimates from forced alignment are off by tens of milliseconds. We test scattering as the detector's input representation. Its evidence is only on synthetic chirps and arpeggios, so whether it helps on speech is an open empirical question for us, not an assumption.
Two works shape the design.
- Diagnose, Then Refine. This shows that utterance-level metrics miss local prosodic defects, and that an audio-language-model judge plus a refiner beats regeneration. Its reliance on a very large judge (Gemini-3-pro) is the latency problem a small detector addresses.
- Semantic Learning for Audio Applications: A Computer Vision Approach. This notes that transients appear as vertical lines on spectrograms. That observation is exactly why a context-free edge operator would flag every plosive, and it motivates our phonetically matched negatives.
Research Program
Training data. We start from natural speech: LibriTTS (LibriTTS) for read prosody, plus the emotional recordings (ESD, RAVDESS) that make up REFINER-DB, for expressive range.
Positives (spliced) are made at word and phone boundaries found by the Montreal Forced Aligner, in four graded families:
- copy-paste from another utterance by the same speaker and emotion;
- copy-paste across emotions;
- the same segment resynthesized by a mel-to-waveform vocoder;
- outputs of existing editors (FluentEditor2-style, FastStitch, VoiceCraft) at their known edit boundaries.
Each positive is paired with negatives. A negative is an untouched natural transition of the same phonetic class: stop-to-vowel, vowel-to-vowel, silence-to-fricative and so on. Negatives are deliberately over-sampled from high pitch-excursion, emphatically stressed regions. As a result, "abrupt" alone never predicts "spliced."
Detector. The detector is a compact two-dimensional convolutional network over a window of roughly 300 ms centred on the nominal boundary, with random jitter of ±50 ms so it cannot rely on exact alignment. The input is either a log-mel spectrogram or scattering coefficients, and the phone classes on either side are supplied as conditioning.
Calibration. We run an active-learning listening study in the style of the JND metric work. Listeners hear a short clip and answer whether it contains a join. Clips are chosen near the detector's decision boundary. The target is a few thousand judgments, an order of magnitude below the 55,000 used for the global JND metric, because the task is narrower. The detector's output is then fit to the listeners' detection probability. The same calibrated detector serves two roles: an evaluation metric reported per seam, and a reranker that scores N candidate edits and keeps the one whose worst seam is least detectable.
Baselines
Naive reference points. These are utterance-level MCD, PESQ, STOI and word error rate, computed as FluentEditor2 and VoiceCraft report them. They represent what the field uses now. Success means our detector's per-edit score correlates with human seam detection and fluency ratings significantly better, tested by bootstrap over listeners. For reranking, the "do nothing" point is the first sample from each editor. The second reference is random seed resampling with the same N, the standard VoiceCraft workflow.
Hand-crafted seam statistics. This is the strongest cheap competitor. We measure spectral flux, energy step, pitch jump and a background-noise-level step across the boundary. Each is z-scored against natural transitions of the same phonetic class, and the scores are combined by logistic regression on the same calibration judgments. It is fit on identical data, so it differs from our detector only in learned versus engineered features.
Global perceptual metric applied locally. We take the JND-trained metric of A Differentiable Perceptual Audio Metric and apply it to the seam window. Where no clean reference exists, it is given an approximate reference, the nearest natural version of the same words. It is then recalibrated on our judgments.
Audio-language-model judge. This follows Diagnose, Then Refine: a prompted audio-language model asks whether and where there is an unnatural join. It runs on the same clips. We report accuracy and wall-clock cost per clip.
Strongest existing editing system. The strongest editing number in the file is FluentEditor2 on VCTK, with a fluency score (FMOS) of 4.314 for insertion against 4.428 for ground truth, and MCD of 5.836 (FluentEditor2). VoiceCraft reports that listeners preferred its edits over the original recording 48% of the time on REALEDIT (VoiceCraft). For the reranking comparison, Diagnose, Then Refine reports that its judge-guided refinement beat regeneration from scratch 77% of the time. For any seam-local objective metric's agreement with listeners, no reported result was found on Lacuna, so that comparison will be established here.
Ablations isolating the key ingredient. The first removes the phonetically matched and high-expressivity negatives, training against random natural windows instead. The second removes phone-class conditioning. The third swaps scattering for log-mel input. If removing the matched negatives does not raise false alarms on stressed plosives and pitch peaks, the central claim is wrong.
Why not just…?
Why not just cross-fade the seam and skip the detector? Cross-fading only fixes amplitude steps at the waveform scale. It cannot fix pitch, coarticulation or room-tone mismatches. The naive splice baseline in Context-Aware Prosody Correction reached 2.87 mean opinion score (MOS) against 4.56 for natural recordings. More to the point, a detector is about measurement, not repair. We will include cross-faded splices among the positives and report whether listeners and the detector still flag them.
Why not just use hand-crafted seam statistics? This is a real threat. Spectral flux and pitch jumps, normalized per phonetic class, may capture most of what listeners hear. Our experiments compare the two directly on the same calibration data. We will report a simple, honest win for the hand-crafted version if it matches the learned detector. We expect the learned detector to pull ahead mainly on expressive and in-the-wild REALEDIT audio, where noise and reverberation corrupt the hand-crafted features. If it does not, the deliverable becomes the normalized statistic plus the human seam dataset, which would still fill the measurement gap.
Why not just use an audio-language-model judge? Diagnose, Then Refine shows such judges catch local prosodic errors well, but at a cost that rules out streaming or wide reranking. It also labels phenomena in coarse binary categories. We will measure both accuracy against our human seam labels and cost. A detector that reaches comparable seam-detection accuracy at a small fraction of the per-clip cost justifies itself. If the judge is far more accurate, the detector's role shrinks to a fast pre-filter, and we will say so.
Why not just add a boundary loss to the generator instead? FluentEditor2 already does this. Without a seam-level metric, nobody can tell whether such losses remove audible seams or only reduce MCD. The detector is what would let that question be answered. A training-loss use is deliberately deferred until the metric is validated.
Proposed Experiments
Validity as a metric. We collect human seam-detection and fluency ratings on edits from FluentEditor2-style diffusion editing, FastStitch and VoiceCraft. These come from VCTK and LibriTTS insertions and replacements, the REALEDIT in-the-wild set, and expressive splices built from ESD and RAVDESS. We then report correlation with human ratings for every baseline above, separately per domain. The decisive check is the expressive subset. Natural emotional utterances with large pitch excursions must score as seamless, while cross-emotion splices of the same words score as seamed. A detector that fails this is simply reacting to abruptness.
Generalization. We train on LibriTTS-derived data and test on VCTK, LJSpeech and REALEDIT, the cross-domain direction where FluentEditor2 reports performance drops. Robustness to alignment error is tested by jittering the nominal seam position by up to ±100 ms.
Utility as a critic. For VoiceCraft on REALEDIT, we sample N candidates per edit and compare the first sample, random choice among the N, the hand-crafted reranker, our detector, and the audio-language-model judge. Each is scored by blind pairwise listener preference and by preference against the original recording, which VoiceCraft reports at 48% unreranked. We also report compute per edit.
The proposal succeeds if the detector beats utterance-level metrics in agreement with listeners, survives the expressive-negatives check, and lifts reranked preference measurably over random resampling at low cost. It is redundant if the normalized hand-crafted statistics match it everywhere.
References
- FluentEditor2: Text-Based Speech Editing by Modeling Multi-Scale Acoustic and Prosody Consistency
- VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
- FastStitch: Speech editing by hitch-hiking a pre-trained FastSpeech2 model
- A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences
- Mesostructures: Beyond Spectrogram Loss in Differentiable Time–Frequency Analysis
- Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction
- Semantic Learning for Audio Applications: A Computer Vision Approach
- LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
- Context-Aware Prosody Correction for Text-Based Speech Editing