Speech recognizers work poorly for people with dysarthria, a motor speech disorder that blurs articulation and distorts timing. The standard fix is to adapt the model to each user with a few of their own recordings. Those recordings carry a voiceprint, medical markers, and whatever the person said, so any adapted model that leaves the device (to a clinician's server, a backup, or a model-merging service) should come with a formal privacy guarantee. Differential privacy (DP) is that guarantee: it bounds how much the released model can reveal about any single recording. It works by clipping each recording's contribution to a fixed size and adding random noise scaled to that clip. The clip and noise are set for the worst case over all possible inputs (the "global sensitivity"), and they are spread across every trainable parameter. For typical speakers this costs a few points of word error rate. For atypical speakers we expect the cost to be worse, because their recordings produce the largest, most unusual gradients. Those are exactly the updates a population-tuned clip truncates most, and the noise then lands in directions that carry no signal for that speaker.
This note proposes to measure that penalty directly and to remove it with two changes. First, project each recording's gradient onto a low-dimensional subspace estimated from public speakers of the same severity group, so that noise is added in $k \approx 25$–$100$ dimensions rather than all adapter dimensions. Second, set the clipping bound per speaker, privately, inside that subspace, so that a severely impaired speaker's informative utterances are not cut down to a bound calibrated on healthy read speech. In two sentences: private speaker adaptation currently sizes its noise to the population and to the full parameter count. We size it to the speaker's own low-dimensional gradient geometry, and we test whether this closes the extra privacy penalty paid by the speakers who most need adaptation.
Core Claim
At a fixed privacy budget, the gap in word error rate (WER) between private and non-private speaker adaptation grows with impairment severity under globally calibrated DP. We claim that most of that excess gap comes from two sources. One is noise spent in gradient directions that the speaker's adaptation never uses. The other is clipping bias from a population-level bound. A severity-matched public subspace plus a per-speaker private clipping bound should recover most of the gap on the Very Low and Low intelligibility groups of UASpeech, and should help little on control speakers. The new elements are the severity matching and this predicted interaction with severity, not projection or adaptive clipping by themselves.
The claim joins two findings that have not been connected. When Does Differentially Private Learning Not Suffer in High Dimensions? shows that fine-tuning gradients concentrate in a small subspace, with singular values decaying roughly as $k^{-0.6}$ on SST-2. In that regime private learning depends on an effective dimension rather than the raw parameter count; retraining in the top 100 directions recovered 90% versus 92% non-private accuracy. On the speech side, Use of Speech Impairment Severity for Dysarthric Speech Recognition and Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition show that dysarthric and elderly variation is captured by low-dimensional spectral bases. The first uses a 25-dimensional SVD-derived speaker embedding and a severity-factored hidden-unit scaling. The second uses SVD-based spectral basis GANs. Li et al. supply the theory that motivates projection. Geng et al. supply both the adapter architecture we reuse and the evidence that severity is a meaningful axis of shared structure. Neither paper considers privacy for atypical speech.
Relation to Existing Private Speech Work
Three papers already apply DP to large speech models, and each fixes its sensitivity in advance on typical speech. Differentially Private Parameter-Efficient Fine-tuning for Large ASR Models fine-tunes a 600M-parameter Conformer privately. It finds that training only bias terms (BitFit) beats low-rank adapters, and it reports 4.6% / 8.1% WER on LibriSpeech test-clean / test-other at $\epsilon=10$, versus 1.9% non-private on test-clean. Enabling Differentially Private Federated Learning for Speech Recognition shows that gradient scales differ sharply across layers and fixes this with hand-set per-layer clipping, but it trains one global model. Training Large ASR Encoders with Differential Privacy freezes the highest-variance 1% of parameters, choosing them from a public LibriLight warm-start. All three evaluate on read English and calibrate on healthy speech, and none personalizes. Our proposal keeps their practical lessons: public warm-starts, freezing normalization biases, and treating per-layer scale as important. It changes the calibration source to severity-matched public data and changes the unit of calibration to the individual speaker.
Private and Stable Test-time Adaptation with Differential Privacy constrains our design in a useful way. In vision, per-sample clipping alone improved continual adaptation by up to 4.1%, and DP-Tent at $\epsilon=20$ reached 62.9% on ImageNet-C against 60.8% for non-private Tent. Clipping can therefore act as a regularizer. That means "looser clipping for atypical speakers" is not obviously better, and our ablations must separate the regularization effect from the bias effect. Renyi Differential Privacy of Propose-Test-Release and Applications to Private and Robust Machine Learning supplies Rényi accounting for propose-test-release under subsampling, which we use for the private choice of a clipping bound. It also reports that the method does not scale to coordinate-wise operations in very high dimensions. That limitation is one reason we only make data-dependent choices after projecting to a few dozen dimensions.
Research Program
Setting and privacy unit. The base recognizer is a Conformer (and, as a second backbone, wav2vec 2.0) trained on UASpeech control speakers plus all dysarthric speakers except the target. This data is treated as public. For each held-out target speaker, blocks B1 and B3 are the private enrollment set and B2 is the test set. The protected record is one spoken utterance. UASpeech records each utterance on several microphones simultaneously, so all channel copies of an utterance form a single record; treating them as separate records would silently weaken the guarantee. Each word appears in several blocks. Utterance-level DP therefore protects individual recordings, not the fact that a person said a given vocabulary word, and we state that limit plainly. The released object is the speaker's adapter: the speaker-specific scaling vectors of a Structured LHUC (learned hidden-unit contribution) adapter, a few thousand parameters, following Geng et al.
Stage 1: a measurement study. Run non-private adaptation for every UASpeech speaker. Record per-utterance gradient norms of the adapter and the singular-value spectrum of the stacked per-utterance gradients, grouped by intelligibility. Two quantities answer whether the premise holds. The first is the fraction of a speaker's utterances that a clip tuned on control speakers would truncate. The second is the number of directions needed to capture 90% of gradient energy for each severity group. If severe speakers neither clip more nor concentrate in a subspace, the proposal stops here, and that negative result is itself worth reporting.
Stage 2: severity-matched projection. For each severity group $g$, estimate an orthonormal basis $P_g \in \mathbb{R}^{k\times d}$ from adapter gradients of public speakers in that group. Here $d$ is the adapter dimension and $k \ll d$. The basis comes from leave-target-out data, so the target's records never touch it. During private adaptation each utterance gradient $g_i$ is projected, clipped, summed over the batch $B$, and noised in the subspace:
$$\tilde g = P_g^\top\Big(\sum_{i\in B}\mathrm{clip}_{C_s}(P_g g_i) + \mathcal{N}(0,\sigma^2 C_s^2 I_k)\Big),$$
Here $\mathrm{clip}_{C}$ rescales a vector to norm at most $C$, $C_s$ is the speaker's clipping bound, and $\sigma$ is the noise multiplier set by the accountant. The noise vector now has expected norm about $\sigma C_s\sqrt{k}$ instead of $\sigma C_s\sqrt{d}$. That reduction is where the projection's benefit, if any, must come from.
Stage 3: per-speaker private clipping. Before training, choose $C_s$ by propose-test-release on a counting query. Propose a bound, then privately test that at most a small number $m$ of the speaker's projected gradient norms at initialization exceed it; a count changes by at most one when one record changes. Release the bound if the test passes, otherwise move to the next candidate on a fixed geometric grid. Accounting follows the Rényi analysis of Wang et al., and this step's cost is charged to the total $\epsilon$. Because the test operates on scalar norms in a projected space, the high-dimensional failure that Wang et al. report does not arise. This step is close in spirit to private quantile-based clipping, and the baselines treat it as such.
Baselines
The naive reference is the unadapted base model evaluated on each target speaker. It costs no privacy, and any private method that fails to beat it is not worth deploying. The non-private ceiling is the same Structured LHUC adapter trained without clipping or noise. For context only, the best published non-private UASpeech system, from Geng et al., reports 17.82% overall WER and 51.25% on the Very Low group. That figure comes from a multi-system ensemble with rescoring, so our single-model ceiling will be measured rather than borrowed.
The strongest existing private method is DP-BitFit in the recipe of Liu et al.: bias-only private fine-tuning, frozen normalization biases, and one global clipping bound. Its reported figure is 4.6% / 8.1% WER on LibriSpeech test-clean / test-other at $\epsilon=10$. No reported result was found on Lacuna for private adaptation on UASpeech, TORGO, or DementiaBank, so we will produce that baseline ourselves. We also run the same Structured LHUC adapter under standard DP-SGD, using per-layer clipping as in the Apple federated recipe and a clip tuned on control speakers. This isolates the calibration change from the adapter change. The LayerFreeze baseline of Chauhan et al. picks frozen parameters from healthy public speech, which tests whether a healthy proxy suffices. Finally, a private quantile adaptive-clipping baseline replaces our propose-test-release step with a privately estimated per-speaker norm quantile, applied in the full adapter space.
Fairness rules apply to every method. All methods use the same base model, enrollment data, privacy accountant, budget grid ($\epsilon \in \{1, 3, 8\}$, $\delta$ below the inverse of the enrollment size), and number of passes. All hyperparameters are tuned only on public held-out speakers of matched severity, never on the target's private data, with an equal search budget per method.
Two ablations isolate the key ingredient. The first replaces the severity-matched subspace with a subspace estimated from control speakers, holding everything else fixed. The second replaces it with a random subspace of the same dimension $k$. "Beating" the baseline is defined in advance. On the Very Low and Low groups, the full method must close at least half of the WER gap between global-clip DP and the non-private ceiling at $\epsilon=3$, with speaker-level bootstrap intervals excluding zero. It must also change control-speaker WER by less than the run-to-run noise.
Why not just…?
Why not just run vanilla DP-SGD on a small adapter, since Li et al. show it already adapts to low effective dimension? This is the most serious threat. Li et al. prove that vanilla DP-SGD's excess risk depends on effective dimension without any explicit projection, which could make our projection a no-op. Their analysis is convex, however, and our enrollment sets are a few thousand utterances run for a few epochs, where finite-sample noise still matters. The healthy-versus-severity-matched-versus-none comparison settles this. If vanilla DP-SGD with a per-speaker clip matches the projected version, we will report that the gain is purely calibration and drop the projection claim.
Why not just use private quantile adaptive clipping, with no subspace or propose-test-release? It is also data-adaptive, cheap, and widely used, so it may capture most of the per-speaker calibration benefit. Our propose-test-release step differs mainly in giving an explicit certificate on how many utterances are clipped, which may not matter in practice. The quantile baseline is therefore run both in the full space and inside our subspace. If it matches propose-test-release in the subspace, we adopt the simpler quantile rule and attribute the result to the severity-matched geometry.
Why not just warm-start on synthetic or GAN-augmented atypical speech and skip privacy-sensitive training? Liu et al.'s text-to-speech warm-start produces normative speech that lacks dysarthric articulation. Jin et al.'s strongest augmentation, a DCGAN, needs parallel recordings of the same text, and a GAN trained on the patient's own speech reintroduces the memorization risk. Augmentation also complements our method rather than replacing it: it can improve the public base model and the public subspace estimate. We include a "base model trained with public augmentation, no private adaptation" arm to measure how much of the gap augmentation alone closes.
Why not just tune one global clip on dysarthric public speakers instead of per speaker? This is a cheaper middle ground. It corresponds to the severity-matched subspace with a fixed, publicly tuned group clip, and we run it as an ablation. If it performs as well as the per-speaker bound, per-speaker private calibration is unnecessary, and the recommendation simplifies to calibrating by severity group.
Proposed Experiments
The primary benchmark is UASpeech, with WER reported per intelligibility group. TORGO serves as a second dysarthric corpus and DementiaBank Pitt as an elderly-speech transfer test, both using the same leave-speaker-out protocol. LibriSpeech test-clean and test-other check that the method does not hurt typical speech. Privacy-utility curves over $\epsilon$ are plotted per severity group. The headline statistic is the excess private gap: private WER minus non-private adapter WER, as a function of severity.
Three checks would confirm or refute the proposal. First, under global-clip DP the excess gap must increase monotonically from control to Very Low speakers; without this, the premise fails. Second, the severity-matched subspace must beat both the control-speaker subspace and the random subspace on severe speakers, which is what ties the gain to acoustic structure rather than to dimension reduction in general. Third, an empirical privacy check runs a membership-inference attack on enrollment utterances against the released adapters. Its success rate should not exceed that of the global-clip baseline at equal $\epsilon$, which confirms that the data-dependent clipping choice is properly accounted for. A supplementary diagnostic correlates each speaker's gradient-energy concentration from Stage 1 with that speaker's gain. A positive correlation would indicate in advance which users benefit.
References
- When Does Differentially Private Learning Not Suffer in High Dimensions?
- Use of Speech Impairment Severity for Dysarthric Speech Recognition
- Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition
- Differentially Private Parameter-Efficient Fine-tuning for Large ASR Models
- Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers, and Gradient Clipping
- Training Large ASR Encoders With Differential Privacy
- Private and Stable Test-time Adaptation with Differential Privacy
- Renyi Differential Privacy of Propose-Test-Release and Applications to Private and Robust Machine Learning