Zero-Depth Prosody: Pitch Conversion That Is Linear in the Right Basis, Evaluated Exactly Under Homomorphic Encryption

Homomorphic encryption lets a server compute on data it cannot read. The catch is cost. Adding encrypted numbers, or multiplying them by constants the server knows, is cheap. Multiplying two encrypted numbers is expensive, and anything non-polynomial (logarithms, divisions, comparisons) is very expensive. Speech researchers, meanwhile, have largely given up on the one prosody transform that fits this cost model: linear pitch scaling, which shifts and stretches a speaker's pitch contour to match a target speaker's average and range. They regard it as too crude, because it flattens intonation into robotic speech. Encrypted speech processing has therefore stayed on spectral features. Pitch (the fundamental frequency, $F_0$) is left out entirely, or sent to the server in the clear, even though pitch carries speaker identity.

This note argues that the two communities are talking past each other. What speech researchers found inadequate is a single global affine map. The strongest multi-scale evidence says that a speaker's prosodic identity lives in particular temporal scales of the pitch contour. A transform that rescales those scales separately is still linear in log-pitch. Once folded into one matrix, it costs the encrypted server nothing beyond multiplications by known constants. We propose to measure how much of the gap to non-linear pitch conversion such scale-selective and learned linear maps close. We then run them exactly under the integer homomorphic scheme BFV with a fixed-point encoding that never needs the truncation step these schemes lack, and state plainly the narrow deployment regime where doing this under encryption is worth it at all.

Core Claim

Let $x$ be a speaker's interpolated log-$F_0$ contour over an utterance, represented as a vector of frames. Every prosody transform in this proposal has the form

$$y = A x + b,$$

where $A$ is a banded matrix known to the server and $b$ is a known offset vector. The conversion is described by $A$ and $b$; the audio is not. Global mean-variance pitch scaling is the special case $A = g I$. The selective wavelet transfer of Sisman & Li is also a special case, $A = \sum_s g_s R_s C_s$. There, $C_s$ is the fixed-point convolution that extracts wavelet scale $s$ (a band of temporal detail, from phone-level wiggles to sentence-level arcs), $R_s$ is its reconstruction weight, and $g_s$ is a per-scale gain.

A third case fits a free banded $A$ by least squares on time-aligned parallel utterances. It asks directly whether linearity, rather than globality, is the problem.

Because $A$ is plaintext, evaluating $y$ on an encrypted $x$ uses only plaintext-times-ciphertext products, slot rotations and additions. That is zero ciphertext-ciphertext multiplications. Fixed-point arithmetic, which stores reals as integers scaled by a factor, normally suffers when scales compound under repeated multiplication. Here the scale grows exactly once, from $S_x$ (the input scale) to $S_x S_A$ (input scale times the scale used to round the coefficients of $A$), and the key holder removes it after decryption. There is no encrypted truncation, no bootstrapping and no polynomial approximation.

The main claim is empirical and has two parts. First, scale-selective or learned linear maps recover most of the speaker-similarity and naturalness gap between global linear scaling and non-linear conversion on standard voice-conversion data. Second, they do so at an encrypted cost comparable to a single linear speech transform, with outputs bit-identical to a plaintext fixed-point simulation. If the first part fails, the second is moot, and the plaintext study is designed to find that out first.

Where This Sits in Prior Work

The case against linear prosody motivates the distinction we draw. Zhang & Tao used a linear pitch model for English words embedded in Chinese sentences and cut pitch error relative to regression trees. Their model and the later critiques show it copies the source contour's shape into a new register. Work on singing (Lu et al.) finds linear scaling cannot produce vibrato or portamento. Work on stress alignment (Morley et al.) finds it cannot move pitch peaks relative to stressed syllables. These results constrain the proposal. Any map of the form $Ax+b$ with a fixed, time-invariant $A$ cannot re-time peaks or invent vibrato, so we will not claim singing or cross-lingual stress re-alignment.

Sisman & Li supplies the method we reuse. It decomposes $F_0$ into 10 wavelet scales on VCC 2016 and finds identity concentrated in scales 4–8. Converting only those scales raised pitch correlation from 0.842 to 0.854, and 82% of listeners identified the target speaker. Their implementation is floating-point and embedded in a sparse-representation system. The observation that their selective transfer is a fixed linear operator on log-$F_0$, and can therefore be compiled into one encrypted matrix product, is ours. Du et al. and Diff-HierVC represent the non-linear end: a CycleGAN over wavelet scales and a diffusion pitch generator. They define the quality ceiling we measure against, and neither is plausibly evaluable under integer homomorphic encryption. A further warning from En-Najjary et al. is that converting pitch independently of the spectral envelope creates mismatches. We hold spectral conversion fixed across all conditions so that this effect is equal across systems rather than confounded with them.

On the cryptographic side, Thaine & Penn supplies the encoding recipe. They computed MFCC/BFCC features over BFV with scaling $10^3$, kept Kaldi recognition accuracy on TIMIT essentially unchanged, and paid about 46.5 s for a Fourier transform over 100 frames and about 346 s per encrypted logarithm. We differ in extracting no spectral features and in placing the one non-linear step, the logarithm of pitch, on the client. Chen et al. shows what happens when a workload needs depth: logistic regression over FV required bootstrapping fused with rescaling, and training took 14.9 to 115 hours. Our design is built to need neither. Teixeira et al. ran x-vector extraction with fixed-point secret sharing at about 1% embedding error. Their costs were 10.85–17.97 s with three or four non-colluding servers and over 41 hours in the two-party malicious setting. That is the interactive alternative we compare against. Laur et al. established the older hybrid pattern of additive encryption for inner products plus garbled circuits for non-linear steps. Our computation is an inner product only, so it needs just the first half of that pattern.

Finally, the privacy motivation. The VoicePrivacy baseline anonymizes the speaker embedding but passes $F_0$ to the synthesizer in plaintext. Srivastava et al. show that plaintext conversion is defeated by informed attackers, with speaker-verification error falling to 0.20%. Encryption does not anonymize anything by itself. It protects the pitch contour from the server doing the work, which is a different guarantee, and the note keeps the two apart.

Threat Model and the Regime Where Encryption Earns Its Cost

The honest weakness of any encrypted linear transform is this: if the output returns to the client who holds the input, the client could have applied the transform locally. The client can also recover $A$ by regression from a handful of input-output pairs. We therefore commit to a narrower regime. The converted contour stays encrypted and is consumed by a further party or stage that the transform's owner controls.

The concrete instance is a relay for anonymized calls or voice notes. Sender and recipient share a key. The platform holds the per-session pseudo-speaker transform, meaning the per-scale gains and target offsets. The platform must apply that transform without hearing the speech. The recipient decrypts only prosody that has already been transformed. The platform learns nothing about the contour. The sender never sees the transform, although a recipient who also held the original could of course fit it.

A second instance is an encrypted analytics pipeline, such as a server-side paralinguistic or verification model. There, speaker-normalizing the pitch is the input layer and its output never leaves ciphertext. We will quantify the regression attack directly, as the number of utterance pairs needed to recover $A$ to a given accuracy. That way the limits of parameter secrecy are measured rather than asserted.

Research Program

On the client, pitch is tracked with the WORLD vocoder. Unvoiced gaps are interpolated as in the wavelet-prosody literature, and the contour is converted to log scale and normalized by the client's own mean and standard deviation. Doing this on the device removes the logarithm and the square-root/division steps. Those are exactly the operations that cost 346 s in Thaine & Penn and dominated Teixeira et al.'s pipeline. The normalized contour is scaled by $S_x$, rounded, packed one frame per slot, and encrypted under BFV in Microsoft SEAL. The voiced/unvoiced mask travels encrypted alongside it, untouched.

On the server, $A$ is banded because wavelet filters and learned alignment maps have finite support. The product $Ax$ is therefore evaluated as $K$ diagonals, where $K$ is the band width. Each diagonal costs one rotation of the encrypted vector, one multiplication by a plaintext vector of rounded coefficients, and one accumulation, following the diagonal method for plaintext matrix times encrypted vector. The offset $b$ is added as a plaintext. Longer utterances span several ciphertexts, with the band overlapping at their boundaries.

The precision analysis is short, which is why the design is attractive. The output error is bounded by the triangle inequality over the $K$ taps:

$$|\hat{y}_i - y_i| \le \frac{\|A_{i,:}\|_1}{2S_x} + \frac{K\,\|x\|_\infty}{2S_A}.$$

Here $\hat{y}_i$ is the decrypted fixed-point output for frame $i$, $y_i$ is the floating-point output, $\|A_{i,:}\|_1$ is the sum of absolute coefficients in that row of $A$, and $\|x\|_\infty$ is the largest input magnitude. The first term is input rounding carried through the row. The second is coefficient rounding accumulated over $K$ taps. For a given perceptual tolerance in cents, this bound fixes $S_x$ and $S_A$. The largest possible integer, $S_x S_A K \|x\|_\infty$, then fixes the plaintext modulus, which in turn fixes the smallest secure ring dimension. We will verify empirically that no wrap-around occurs.

The three transforms (global, selective-wavelet, learned banded) are fitted in plaintext on VCC 2016 training data. For the learned map, frames are aligned by dynamic time warping and the banded least-squares problem includes a ridge penalty.

Baselines

The naive reference point is global linear pitch scaling, the $A = gI$ special case, run through the identical client, encryption and resynthesis path. It is the method the literature rejects, so our wavelet and learned maps must beat it clearly. Otherwise nothing is gained over what is already known to be inadequate. The file has no reported VCC 2016 number for plain global scaling. The closest figure is Zhang & Tao's 41.88% pitch-error reduction over regression trees, on a different bilingual corpus. We therefore rerun it ourselves.

The strongest existing linear-family method is Sisman & Li's selective transfer of scales 4–8. They report pitch correlation 0.854 and 82% correct target identification on VCC 2016. We reimplement it in floating point and must first reproduce those numbers within noise before comparing. Our learned banded map counts as beating it if it raises target identification in the same XAB-style test (listeners judge which reference a sample sounds like) with significance, at equal or better naturalness.

The non-linear ceilings are the wavelet CycleGAN of Du et al. and diffusion pitch generation from Diff-HierVC. Du et al. report MOS 2.92/3.34, but only in a cross-lingual setting, so it is not directly comparable. Diff-HierVC reports EER 3.29% and CER 0.83% on LibriTTS/VCTK. We run both pitch modules on VCC 2016 with the same spectral conversion and vocoder, and the same training utterances and tuning budget as our fitted maps. For the diffusion model, only the pitch generator is swapped in, to keep the spectral path fixed. "Closing most of the gap" is defined in advance as recovering at least half of the XAB and MOS difference between the global-linear and best non-linear system.

On the cryptographic side, the reference points are three. The first is plaintext floating point. The second is plaintext fixed-point simulation, which the encrypted result must match bit-for-bit. The third is the same linear map under CKKS, the approximate scheme with built-in rescaling. As an interactive comparator we evaluate the identical map under three-party replicated secret sharing in MP-SPDZ, the setup of Teixeira et al..

The ablation that isolates the key ingredient keeps everything fixed and varies only the basis in which linearity is imposed: global, wavelet-selective, or learned banded. It also includes a learned diagonal-in-wavelet map, which separates "choosing scales well" from "arbitrary banded filtering."

Why not just…?

Why not just apply the pitch transform on the client in plaintext? In any deployment where the output returns to the sender, this is strictly better, and we concede it. The linear map is cheap, and the client can recover it by regression anyway, which we will measure. The proposal is only worth doing in the relay and encrypted-pipeline regimes, where the transform's owner must apply it without seeing the contour and the output goes to someone other than the sender. The plaintext study also stands on its own: whether linear-in-the-right-basis maps are good enough is an open speech question regardless of encryption.

Why not just use secret sharing instead of homomorphic encryption? For a public linear map on secret-shared data, secret sharing is essentially free, so this is a real competitor. Teixeira et al., however, needed three or four non-colluding servers to be fast, and over 41 hours in the two-party malicious setting. When the platform is a single operator that holds $A$ privately, the computation becomes a two-party problem with private matrix and private vector, and additively homomorphic evaluation is the natural tool. We run both on identical inputs and report latency and communication. If three-party secret sharing wins on latency under a plausible non-collusion assumption, we will say so.

Why not just use CKKS, which handles real numbers natively? For a zero-depth computation CKKS would likely work fine and may be faster. The case for exact BFV is that its outputs are bit-reproducible. Li et al. also show that the approximation noise in CKKS decryptions can leak the secret key to passive observers unless extra noise is added. In our relay regime the recipient decrypts and might pass results onward, so this matters. We run both, report accuracy and cost, and treat CKKS with noise flooding as a legitimate option if its cost is lower at equal error.

Why not just evaluate a small non-linear pitch model under encryption? Even a shallow network requires ciphertext-ciphertext multiplications and polynomial activations. Chen et al. shows this pushes simple models into bootstrapping and hours of compute. We include one polynomial-activation two-layer pitch model under BFV as a cost probe. It does not have to lose on quality; it has to cost far more per utterance, which is what the probe will measure.

Why not just anonymize in plaintext and skip cryptography? Plaintext anonymization and encryption protect against different parties. Srivastava et al. shows plaintext conversion fails against informed attackers. Our pipeline does not fix that for the listener; it only keeps the server blind. We measure listener-side identity leakage separately, with a verification attack, so the two guarantees are never conflated.

Proposed Experiments and Decisive Checks

The plaintext study uses VCC 2016 (The Voice Conversion Challenge 2016): 16 speaker pairs, 162 training and 54 evaluation utterances per speaker. Each speaker's WORLD-extracted log-$F_0$ is converted by each baseline, then resynthesized with a single fixed spectral conversion. We report pitch RMSE and correlation, both overall and per wavelet scale, together with a listening test on target identification and naturalness MOS with confidence intervals.

The first decisive check is simple. If the wavelet-selective or learned banded map does not close at least half the gap between global linear and the best non-linear system, we stop the encryption work and publish the negative result as a quantified boundary on linear prosody.

The fixed-point study sweeps $S_x$ and $S_A$ in bits and compares the measured error with the bound above. It also locates the smallest scale at which listeners cannot tell fixed-point output from floating-point output in an ABX test (listeners judge whether a sample matches one of two references). The encrypted study runs the chosen maps in SEAL BFV at the smallest secure parameters the bound permits. We report per-utterance server latency, ciphertext size and noise budget remaining. For scale reference, Thaine & Penn's roughly 46.5 s for a 100-frame encrypted Fourier transform is the only encrypted linear speech-transform figure in the file.

The second decisive check is that the decrypted output equals the plaintext fixed-point simulation exactly on every evaluation utterance. Any mismatch indicates modulus overflow or an encoding bug.

The privacy study has two measurements. On LibriSpeech with the VoicePrivacy protocol, we measure how much speaker identity the pitch contour alone carries by running speaker verification with and without mid-scale conversion, under both ignorant and informed attackers. This separates the listener-side effect of the transform from the server-side guarantee of encryption. We also report the number of parallel utterance pairs a party holding both sides needs to recover $A$ to within a stated error, which bounds how much parameter secrecy the relay design can actually offer.