Choosing Where to Inject Speaker Embeddings Without Training: A Gradient-Covariance Score

Speech recognizers do better when they are told who is speaking. The usual way is to compute a short "voiceprint" vector for the speaker, such as an i-vector or x-vector (a fixed-length summary of a speaker's voice and recording channel), and feed it into the recognizer. Three choices remain open: where in the network to feed it, how to combine it with the hidden activations, and which kind of voiceprint to use. Today these are settled by training many variants to convergence and keeping the best. One paper tried every encoder block of wav2vec 2.0 before settling on the second (Speaker adaptation for Wav2vec2 based dysarthric ASR). Another hard-coded x-vectors into Whisper's decoder (Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition). A third found that the best choice flips between early and late injection depending on noise level (Speaker Adaptation for End-To-End Speech Recognition Systems in Noisy Environments). Each new population, such as a new dysarthria etiology, a new accent, or a new noise regime, restarts this expensive trial and error.

We propose to replace that search with a score computed from one forward and backward pass of the already-pretrained model. Suppose a speaker-conditioned shift is added at some layer and initialized to zero, so the network's output does not change at all. A first-order expansion then tells us how much the loss would drop after one optimal step. That drop equals how strongly the per-utterance loss gradients at that layer can be predicted from the speaker embedding. Layers, combination operators, and embedding types can then be ranked by this "speaker-gradient covariance" in minutes rather than GPU-days. Because the score measures speech-recognition benefit directly, it should also explain a known puzzle: embeddings that are best at telling speakers apart are not the ones that most help recognition (Embeddings for DNN Speaker Adaptive Training).

Core Claim

For a pretrained acoustic model, the usefulness of injecting a speaker embedding at a given layer with a given operator can be predicted without any training. The predictor is the share of the layer's loss-gradient variance that is linearly explained by the embedding. We claim this score ranks candidate configurations in the same order as their converged word error rate (WER), well enough to find a configuration within a small margin of the best one. We also claim it does so at a cost one to two orders of magnitude below short fine-tuning of each candidate.

This is a different object from existing zero-cost proxies. EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring evolves scoring formulas that are computed at random initialization on vision cells. The research file notes that such proxies assume Conv-BatchNorm-ReLU blocks and do not capture conditioning or cross-branch wiring. Our score is computed on a trained backbone, and it is specific to an auxiliary input. A generic proxy cannot even see the embedding when the adapter starts at zero.

The Score

Let $h_l(u)$ be the time-averaged hidden activation at layer $l$ for utterance $u$. Let $g_l(u) = \partial \mathcal{L}(u) / \partial h_l(u)$ be the gradient of the recognition loss (CTC or sequence cross-entropy) with respect to that activation, averaged over frames. Let $e_u$ be the speaker embedding.

Consider an additive shift $\beta(e) = A e$ inserted at layer $l$ with $A = 0$, which preserves the network's function in the same spirit as FNA's identity-preserving remapping (Fast Neural Network Adaptation via Parameter Remapping and Architecture Search). To first order, the loss change is $\sum_u \langle g_l(u), A e_u \rangle$. The steepest-descent direction for $A$ is $G_l^\top E$, where the rows of $G_l$ are the per-utterance gradients and the rows of $E$ are the embeddings, both centered across utterances. After whitening the embeddings, so that the score reflects the information they carry rather than their scale, the predicted improvement is

$$S_l^{\text{add}} = \big\| G_l^\top E\,(E^\top E)^{-1/2} \big\|_F^2 .$$

This is the gradient energy recoverable by linear regression on $e$. Centering $G_l$ removes the part of the gradient that any speaker-independent bias could absorb, so the score isolates speaker-specific benefit.

A multiplicative gate $\gamma(e) = 1 + B e$ gives the same expression with $g_l(u) \odot h_l(u)$ in place of $g_l(u)$. That lets the score arbitrate the additive-versus-scaling question that Embeddings for DNN Speaker Adaptive Training settled empirically in favour of additive shifts.

Swapping $E$ for i-vectors, x-vectors, ECAPA-TDNN embeddings, or concatenations scores embedding types on the same footing. The whole computation needs one backward pass over a few hundred adaptation utterances.

Research Program

Backbones and search space. We will use wav2vec 2.0 Base and Large and the Whisper-large-v3 encoder and decoder. Each candidate configuration is defined by three things: the insertion point (each encoder block, split into post-attention and post-feed-forward sublayers), the operator (additive, multiplicative, or both), and the embedding type (i-vector, x-vector, ECAPA-TDNN, and fMLLR-plus-x-vector, following the dysarthric wav2vec work). We restrict the space to affine operators on purpose, because these are the ones for which the first-order derivation is exact. Cross-attention fusion of the kind used in Qifusion-Net: Layer-adapted Stream/Non-stream Model for End-to-End Multi-Accent Speech Recognition enters as an external comparison rather than as a scored candidate.

Ground truth. On a 12-layer backbone the single-point space has roughly 24 sublayers × 3 operators × 4 embeddings, about 288 configurations. We will train a stratified sample of about 100 of these to convergence per dataset. This expensive ground truth is the scientific contribution in its own right: no such table exists for speaker conditioning. Every proxy is judged against it.

Multi-point selection. We will select layers greedily with residualization. After choosing a layer, we refit its regression and project the explained component out of the gradients at the other layers before rescoring. This tests whether the score avoids redundant placements, which matters because dense all-layer conditioning brought no gain on AMI (Embeddings for DNN Speaker Adaptive Training).

Condition-specific scoring. We will compute the score separately for each noise level and each etiology subgroup. The goal is to see whether one cheap measurement predicts that the best configuration changes with the regime.

Baselines

Unadapted backbone. The naive reference is the same backbone fine-tuned identically with no speaker input. Its role is to calibrate how much speaker conditioning is worth at all on each dataset.

Best hand-designed method. The strongest existing configuration on each benchmark is the published hand-chosen one, which we re-run on our splits:

Proxy-selected configurations get the same fine-tuning recipe, data, and number of tuning trials as these baselines. Beating a baseline means matching or improving its WER while spending far less on search, or improving WER outright at equal final training cost.

Budgeted short fine-tuning. The strongest search-cost competitor fine-tunes every candidate briefly and ranks by validation WER. Hyperparameters are chosen by the minimum-importance pruning procedure of Rethinking Performance Estimation in Neural Architecture Search so that this baseline is as strong as possible. We will sweep its budget from a few hundred steps to one epoch and plot rank correlation against GPU-hours, with our score as a single point on the same plot.

One-shot search. We will also run a Gumbel-Softmax placement-and-operator search in the style of Adaptable Adapters, trained once and given the same total compute.

Generic zero-cost proxies. We will include Synflow, NASWOT, and EZNAS-A applied to the adapter-augmented network. There is no reported result on Lacuna for any of these on speech; the only file figure is EZNAS-A's Kendall τ of 0.65 on NAS-Bench-201 CIFAR-10.

Ablations of the key ingredient. Three ablations isolate what the score depends on:

  • Shuffled embeddings: randomly permute $e_u$ across speakers. The score should collapse. If it does not, it is measuring gradient magnitude rather than speaker information.
  • Uncentered gradients: this tests whether speaker-independent bias would masquerade as speaker benefit.
  • Plain gradient norm: replace the score with $\|G_l\|_F^2$, which tests whether the embedding term adds anything beyond "large gradients here."

Why not just…?

Why not just fine-tune each candidate for a few hundred steps? This is the real threat. The space is small enough that brute-force short fine-tuning is feasible, and if it ranks as well at comparable cost our contribution reduces to a cheaper but equal shortcut. Two pieces of evidence argue against it. First, short-training ranking is known to be fragile: Rethinking Performance Estimation in Neural Architecture Search found that the number of training epochs is one of the hyperparameters most critical for ranking fidelity. Second, freshly inserted adapters move slowly at first, which should compress early differences between candidates. The budget-versus-correlation curve settles the question. If short fine-tuning reaches the score's Kendall τ at under five times its cost, we will report that the score is only a convenience. In that case we will reposition it as an initializer for short fine-tuning, used to prune the space before training.

Why not just condition every layer with learned weights, as Qifusion-Net or a one-shot supernet does? All-layer weighting sidesteps the placement question. However, Embeddings for DNN Speaker Adaptive Training found that a multi-layer control network was no better than a single linear input shift. Rethinking Performance Estimation in Neural Architecture Search shows that one-shot supernets often lose global ranking fidelity once candidates are trained in isolation. We run both as baselines at matched compute. If all-layer conditioning matches the proxy's choice in WER, the remaining argument for the score is parameter count and diagnostic value, and we will say so.

Why not just pick the embedding with the best speaker-verification accuracy? Verification accuracy is the obvious cheap selector, but Embeddings for DNN Speaker Adaptive Training reports that it does not track recognition gains: channel-aware i-vectors beat x-vectors. We will measure the rank correlation of verification equal-error-rate with converged WER alongside that of our score. The prediction is that verification accuracy is near zero and our score is clearly positive.

Why not just use a bigger model? Scale is orthogonal rather than a substitute. Whisper-large-v3 still sits at 12.31 development WER on SAP before personalization (Personalized Fine-Tuning with Controllable Synthetic Speech from LLM-Generated Transcripts for Dysarthric Speech Recognition). We will test the score on both Base and Large backbones to check that its ranking holds across scale.

Proposed Experiments and Decisive Checks

Datasets.

  • AMI IHM (100 h) for meeting speech.
  • LibriSpeech with MUSAN noise at 18, 9, and 0 dB, following the noisy-adaptation protocol. Its Transformer baseline improved from 51.87% to 46.95% WER at 0 dB on test-clean when speaker embeddings were added (Speaker Adaptation for End-To-End Speech Recognition Systems in Noisy Environments).
  • UASpeech, with leave-one-speaker-out splits by severity.
  • SAP, with results broken down by etiology.

Metrics. We report three measures: Kendall τ between the score and converged WER over the ground-truth configurations; top-5 recall; and regret, defined as the WER of the selected configuration minus the oracle WER. Search cost is reported in GPU-hours.

Decisive checks. Five outcomes decide whether the proposal holds:

  1. Ranking quality. The score must reach a Kendall τ at least equal to short fine-tuning at its cost-matched budget, and regret must stay under 0.5 absolute WER on AMI and LibriSpeech.
  2. Embedding ranking. Without any training, the score must order i-vectors above x-vectors on AMI, reproducing the earlier finding.
  3. Noise flip. Computed separately at 0 dB and at 18 dB, the score must move its preferred insertion point earlier as noise increases. This is the early-versus-late reversal that was previously discovered only by training both variants.
  4. Dysarthric speech. On UASpeech, the proxy-selected configuration must match or beat the hand-chosen second-block adapter.
  5. Etiology-specific selection. On SAP, configurations selected per etiology must narrow the gap between the best and worst groups, for example between ALS and Down syndrome.

Failure criterion. If the shuffled-embedding ablation scores within noise of the real one, or if τ does not exceed the plain gradient-norm control, we will conclude that first-order speaker-gradient covariance does not capture conditioning utility.