Speaker verification systems decide whether two recordings come from the same person. They work well on 30-second recordings and break down on half-second ones: a short clip contains only a few sounds, so its "voiceprint" (a fixed-length vector called a speaker embedding) is dominated by which words happened to be spoken rather than by who spoke them. The current fixes learn to turn a short-clip embedding into a long-clip one, or to generate the missing speech. All of them are trained on pairs of short crops and the long recordings they were cut from. On VoxCeleb such pairs are free. In the domains where short-utterance verification is actually needed, they often do not exist. Wake-word devices, short telephone turns and prompted digit strings yield only short clips, and nobody holds long, clean, in-domain recordings of the same speakers.
We propose to train the short-to-long embedding compensator using only short clips. The method borrows the measurement-splitting trick from MRI reconstruction without ground truth. The clips from one recording session are cut into two disjoint time portions. A small network learns to predict the frozen backbone's embedding of one portion from the embedding of the other. Because the "word noise" in the two portions is roughly independent, the network cannot learn to reproduce it. What it can learn is the speaker-and-session part of the embedding, which is what a long recording would have revealed. In two sentences: duration truncation is treated as a measurement process, and the long-utterance embedding is the unobserved clean signal. A compensator trained to predict one short crop's embedding from another, disjoint crop of the same session learns a clean-signal estimator without ever seeing a long recording.
Core Claim
The claim is stated in one place here. Let $x$ be the acoustic features of a session from which only short clips are available, and let $x_\Theta$ and $x_\Lambda$ be two disjoint time portions of it, separated by a small guard gap. Let $e(\cdot)$ be a frozen, pre-trained embedding extractor with unit-normalized output, and $d_\Theta$ the duration of $x_\Theta$. We train a lightweight compensator $f_\phi$ with
$$ \mathcal{L}(\phi) = \mathbb{E}\,\big\| f_\phi\big(e(x_\Theta), d_\Theta\big) - e(x_\Lambda) \big\|^2 . $$
Assume the deviations of $e(x_\Theta)$ and $e(x_\Lambda)$ from their session means are independent given the session $s$. Then the minimizer is the posterior mean of $\mu_{d_\Lambda}(s) = \mathbb{E}[e(x_\Lambda)\mid s]$ given the short observation. The same Noise2Noise-style argument underlies the k-space splitting of Yaman et al.. A paired, supervised compensator would instead target $\mu_\infty(s)$, the long-recording embedding.
The method is therefore exact up to one measurable quantity: the duration bias between $\mu_{d_\Lambda}$ and $\mu_\infty$. Because embeddings are nonlinear and normalized, short-crop means are shrunk and possibly rotated relative to long-crop means. A large part of the research is to measure this bias as $d_\Lambda$ grows and to keep it small. We do this by assembling $\Lambda$ from all remaining material in the session rather than from a single short crop.
What is new is not "denoise embeddings". It is the claim that the paired long-utterance data every compensation method relies on is unnecessary. The claim comes with a precise statement of what is lost, and a test of whether the loss matters.
Relation to Prior Work
Three lines of short-duration work motivate the gap by depending on long references. Zhang et al. train a conditional Wasserstein GAN that maps short-clip i-vectors to their paired long-clip i-vectors. VoiceExtender trains a speaker-guided diffusion model on long utterances to synthesize 1.5 s of extra filterbank frames. Zhao et al. prompt zero-shot speech synthesizers pre-trained on massive transcribed corpora. We keep their goal of recovering what a longer recording would show, but drop the long references. We also drop the generation step, which is slow at test time and, by Zhao et al.'s own finding, drifts in identity: synthetic-only embeddings are worse than the original clip, so they must interpolate with weight about 0.6 to 0.8 on the original.
Two lines of work supply the method. Bousquet and Bonastre cut enrollment audio into short partitions and average their i-vectors, and separately estimate a short-utterance variance matrix from full-versus-truncated pairs. We use partitions as a training signal rather than an averaging heuristic, and we need no full-length counterpart. Measurement splitting from Yaman et al. and its scan-specific variant (Yaman et al.) supplies the core loss: train on one subset of measurements and score against a disjoint subset with independent noise.
One limitation in the inverse-problems literature constrains the design. Splitting cannot recover a component that no split ever observes, which is why Equivariant Splitting adds symmetry assumptions. In our setting the missing component is the duration bias above, and we measure it directly rather than assume it away. Those methods also assume an analytic forward operator. We need only the conditional independence of crop noise, which is why no equivariance, unrolled data-consistency layers, or test-time adaptation in the style of AdaptNet appear in the design.
Architectural work such as ERes2NetV2 and the hybrid-loss attentive-pooling systems of Kwon et al. is complementary. Those systems are the frozen extractor $e$, not competitors to $f_\phi$.
Research Program
We build three things.
The first is the compensator. It is a two- to four-layer residual network on the embedding, conditioned on log-duration, with under a million parameters, so test-time cost is negligible. It is trained on target-domain short clips grouped by session. The session grouping is the only side information assumed: it says which clips came from one recording, not who the speaker is, and no cross-session identity labels are used. A weaker variant uses no grouping at all and splits a single short clip into its own two halves.
The second is a duration-bias study, run on VoxCeleb1 where long ground truth exists. We measure the cosine between $\mu_{d}(s)$ and $\mu_\infty(s)$ as $d$ grows from 0.5 s to 20 s. From this we choose how $\Lambda$ is assembled: a single crop, the concatenation of all other clips, or the average of their embeddings. Averaging is the partitioning idea, repurposed as a target. We also test a correction that renormalizes the predicted vector, since cosine scoring discards norm and shrinkage may then be harmless.
The third is an independence audit. We sweep the guard gap between $\Theta$ and $\Lambda$ and compare within-clip splits to cross-clip splits. This quantifies how much shared coarticulation and lexical content violate the independence assumption, the analogue of choosing the split ratio in Yaman et al.
Baselines
The naive reference point is the frozen extractor, an ECAPA-TDNN trained on VoxCeleb2 and also an ERes2NetV2 extractor, scoring raw short-clip embeddings by cosine. Its tiled-input variant loops the short clip to 3 s; VoiceExtender reports that repeating a 0.5 s clip gives no gain. Every compensator must beat this, at each duration, to justify itself.
The oracle is a paired supervised compensator: the same network trained with $\mu_\infty$ targets cut from long recordings of the same domain, with identical architecture, data volume and tuning budget. On VoxCeleb1 this is the ceiling. The headline result there is the fraction of the oracle's EER reduction that the self-supervised compensator recovers, in the way Yaman et al. compare against supervised MRI training and Equivariant Splitting reports 27.45 dB against a supervised 28.46 dB on DIV2K.
The strongest practical competitor is the source-trained paired compensator. It is trained on VoxCeleb crops, where pairs are free, and applied unchanged in the target domain. It is described in the next section because it is the main threat.
The strongest published short-duration systems are reported as reference points and, where code allows, rerun on our ECAPA extractor and trial lists. On truncated VoxCeleb1, VoiceExtender reports EER falling from 17.29% to 9.32% at 0.5 s and a 35.7% relative reduction at 1.0 s. The zero-shot synthesis approach reports 12.82% to 10.77% at 1.0 s. ERes2NetV2 reports 1.48% EER on 2 s segments. On NIST SRE 2008 10sec-10sec, D-WCGAN score fusion reports 8.80% EER and partitioning with short-utterance-variance PLDA reports 12.05%. On SRE 2010 10sec-10sec the latter reports 11.58%. On RSR2015 Part III, phone-centric local variability vectors report 4.192% (male) and 5.407% (female) EER, and HMM shared-alignment rescoring reports 1.01% and 2.51%. Those figures used different backbones, so beating our rerun matters more than beating the printed number.
The key ablation replaces the nonlinear $f_\phi$ with a linear-Gaussian posterior mean whose short-utterance covariance is estimated from the same splits. This is short-utterance variance modelling without long data. It isolates whether the gain comes from the splitting signal or from network capacity.
Why not just…?
Why not just train the compensator on VoxCeleb crops, where long references are free, and transfer it? This is the real threat, and we treat it as the main comparison rather than a straw man. Its weakness is that the posterior mean it learns depends on the source speaker population, channels and phonetic content. Telephone 10-second segments and digit strings differ from YouTube interviews on all three. Nothing in the file measures how far such a compensator transfers, so the experiment decides. We train it, apply it to NIST SRE and RSR2015, then fine-tune it with our splitting loss on target short clips. If transfer alone matches the fine-tuned version, our contribution shrinks to the in-domain-matched case. We would report that and narrow the claim to domains whose speakers and recording conditions are far from the source.
Why not just adapt the backend without any network: re-center, whiten, or average partitions? Bousquet and Bonastre show that partitioning and variance modelling help substantially, cutting GPLDA error from 15.90% to 12.05% on SRE 2008. Target-domain whitening is cheap. We include both, plus the linear posterior-mean ablation, which is the strongest form of this alternative. If the linear version matches the network, we still report a useful result: short-utterance variance can be estimated from splits alone. The nonlinear compensator would then be unnecessary.
Why not just fine-tune the backbone self-supervised on target short clips, treating two crops of a clip as a positive pair? This adapts the whole extractor rather than a small head. It has known failure modes the regression avoids. Contrastive negatives drawn from other clips can belong to the same speaker, and, because the target $e(x_\Lambda)$ comes from a frozen extractor, the regression has no collapse to guard against. It also touches millions of parameters with little target data. We run it with the same target clips and tuning budget, and report it alone and combined with the compensator.
Why not just extend the clip generatively with diffusion or zero-shot speech synthesis? These methods need generators trained on long, often transcribed speech. They add iterative inference latency. The zero-shot synthesis study reports that synthetic audio alone increases EER and must be blended with the original. We compare accuracy and wall-clock cost per trial on truncated VoxCeleb1, the one domain where their numbers exist.
Proposed Experiments
The controlled experiment uses truncated VoxCeleb1 at 0.5, 1.0 and 2.0 s, with enrollment and test both truncated and VoxCeleb2 as the extractor's training set. The self-supervised compensator sees only short crops grouped by video session. The paired oracle sees long references. We report EER and minimum detection cost.
The two decisive checks are the following. First, the self-supervised compensator should recover most of the oracle's improvement at 0.5 s and 1.0 s. Second, the residual gap should track the measured duration bias between $\mu_{d_\Lambda}$ and $\mu_\infty$, shrinking as $\Lambda$ is lengthened. If the gap does not track the bias, the independence assumption is failing, and the guard-gap audit should show where.
The transfer experiment uses NIST SRE 2008 and 2010 10sec-10sec, and RSR2015 Part III, as target domains in which only short clips are used. Speaker labels are used only for scoring. Here the decisive check is the comparison with the source-trained paired compensator. The target-trained splitting compensator, or that compensator fine-tuned with splitting, must beat it by a margin larger than across-seed variation for the proposal's central claim to hold.
We expect the largest effect at 0.5 s, where VoiceExtender's 17.29% baseline shows the most headroom. We expect little at 2 s, where VoiceExtender's own gain is only 5.7%. A null result at 2 s would therefore not count against the method. A null result at 0.5 s in the transfer setting would.
References
- Self-Supervised Physics-Based Deep Learning MRI Reconstruction Without Fully-Sampled Data
- I-vector Transformation Using Conditional Generative Adversarial Networks for Short Utterance Speaker Verification
- Voiceextender: Short-Utterance Text-Independent Speaker Verification With Guided Diffusion Model
- Investigation of Zero-shot Text-to-Speech Models for Enhancing Short-Utterance Speaker Verification
- Short Utterance Variance Modelling and Utterance Partitioning for PLDA Speaker Verification
- Zero-Shot Self-Supervised Learning for MRI Reconstruction
- Equivariant Splitting: Self-supervised learning from incomplete data
- Test-Time Model Adaptation for Image Reconstruction Using Self-supervised Adaptive Layers
- ERes2NetV2: Boosting Short-Duration Speaker Verification Performance with Computational Efficiency
- The ins and outs of speaker recognition: lessons from VoxSRC 2020