Do Compressed Text-to-Speech Models Inherit Their Teacher's Memorized Recordings?

Speech synthesizers that run on phones, cars and screen readers are almost never trained at their final size. A larger model is trained first and then shrunk by removing weights (pruning), storing them in fewer bits (quantization), or running the generator for only a handful of steps. This note asks whether a recording the large model memorized survives that shrinking. Here "memorized" means a model that, given the training transcript, reproduces the recording's particular timing and pitch rather than a fresh rendition. The privacy literature disagrees on what to expect. One line finds that removing weights strips out the capacity used to fit individual samples. Another finds that tighter capacity lets a model store its training data more efficiently. Neither line has looked at text-to-speech.

We propose to settle the question with planted "prosodic canaries." These are training recordings whose rhythm and pitch were deliberately distorted so that no model could produce them by generalizing from other data. We plant them at controlled repetition counts in a compact flow-matching synthesizer. Then we test which deployment steps erase them and which preserve them. The central comparison is between a model pruned or quantized down to a target size and a model trained from scratch at that same size. The working hypothesis is that compression inherits memorization while small-from-scratch training does not. If so, the usual on-device recipe produces a small model with the leakage of a large one.

Core Claim

In a text-to-speech (TTS) model, the transcript is the conditioning input and almost uniquely identifies a training utterance. The natural extraction attack is therefore simply to type in training sentences. We claim that for a compact flow-matching TTS decoder, how much training prosody can be extracted this way depends mainly on how the deployed model was obtained, not on its final parameter count. Pruned and quantized descendants of a larger model should keep the canaries the larger model memorized. A dense model trained from scratch at the same non-zero parameter count should reproduce them much less.

A secondary claim concerns the knobs used to make sampling cheap. Few-step solvers and guidance (pushing the generator harder toward the text condition) should make memorized prosody more extractable, because they concentrate trajectories. The proposal is mainly a measurement study, and the canary audit is the instrument that makes the measurement possible.

What the Literature Gives Us

Several papers motivate the question. Generation or Replication: Auscultating Audio Latent Diffusion Models prompted a text-to-audio diffusion model with its own training captions and found replication of over 90% of the data when trained on 1,000 clips, versus 5 to 10 copies per 10,000 generations on about 45,000. It studied general audio, not speech, and never touched compression. Membership and Dataset Inference Attacks on Large Audio Generative Models showed that single-sample membership tests are near chance (about 50% AUC) on large audio generators and reach about 70% on TANGO. That finding is the reason we plant canaries rather than rely on membership scores alone. Where Rectified Flows Leak found that in flow models trained on music, the member/non-member reconstruction gap is bell-shaped along the noise-to-data path and peaks at an intermediate point. It also found that this signal keeps growing after validation loss plateaus, which suggests memorization is laid down in the large model's weights early. That paper scaled models up from 140M to 880M parameters, never down, and did not study speech.

Two papers frame the competing predictions. Against Membership Inference Attack: Pruning is All You Need pruned image classifiers with a constraint on the gap between members and non-members and reduced attack accuracy. Parameters or Privacy proved, for linear and random-feature models, that leakage grows with parameters per sample and that pruning trades privacy for utility like added output noise. Both predict that compression helps. Against them, Cram Less to Fit More shows that capacity-limited language models store more facts when they are relieved of redundant data. That supports the view that a smaller network can concentrate rather than lose specific memories. Our pruned-versus-from-scratch comparison separates these views. One caveat matters: the first paper studies discriminative classifiers and the second linear regression, so neither result transfers to generative decoders automatically.

Other papers supply methods. Unintended Memorization in Large ASR Models, and How to Mitigate It planted sped-up utterances as canaries in speech recognition, on the logic that they cannot be learned by generalization. It also showed that clipping the averaged gradient on each hardware core limits memorization of canaries repeated up to 16 times at little cost. We turn the canary idea around for generation, and we reuse per-core clipping as a mitigation baseline. Matcha-TTS provides the compact decoder. It is a 1D U-Net trained with optimal-transport conditional flow matching and samples in 2 to 10 steps. Prior-free Guided TTS provides guidance at no extra inference cost. How Diffusion Models Memorize found that classifier-free guidance drives copying in image diffusion, which motivates our guidance arm.

Research Program

Prosodic canaries. We create a few hundred canaries. Each pairs a unique nonce transcript, a random sequence of real words that appears nowhere else, with a recording of that text read by the corpus speaker or synthesized in-voice. We then distort the recording's prosody in a seeded way. We time-stretch a seeded subset of words by factors between 0.5 and 2, and we impose a seeded pitch contour such as a fixed rise-fall pattern unrelated to the syntax.

Each canary's seed comes from a known family. That lets us compute an exposure-style rank: how close the generated phone durations and pitch contour are to the true seed, compared with a few hundred unused seeds from the same family. Canaries are inserted 1, 2, 4, 8 or 16 times into LJSpeech training. A held-out set of canaries is never inserted and gives the null distribution of ranks.

Natural-data extraction. For real LJSpeech sentences we compare members against held-out sentences. For each, we measure how close the generated phone durations and pitch contour, after alignment, are to the original recording. We also compute mel-spectrogram similarity after dynamic time warping, following the copy-detection design of the audio diffusion study.

Compression arms. All arms descend from one Matcha-TTS teacher trained on the canary-augmented LJSpeech. They are magnitude pruning to 50%, 75% and 90% sparsity with fine-tuning on the same data; post-training quantization to 8 and 4 bits; and distillation into a narrower decoder. Each is paired with a dense model of matched non-zero parameter count trained from scratch on identical data with identical step budgets. All arms are sampled at 2, 4 and 10 solver steps.

Low-data voices. A LibriTTS arm repeats the study per speaker, adapting the teacher to speakers with only a few minutes of speech. This mimics custom on-device voices, where both parameters and data are scarce.

Baselines

Naive reference. The do-nothing reference is the uncompressed teacher, sampled at its default step count, with no mitigation and audited only by its validation loss. It tells us how much memorization exists before any deployment step. It also shows what a developer would see without an audit: Matcha-TTS reports a Word Error Rate of 2.09% and an MOS of 3.84 on LJSpeech (Matcha-TTS). Neither number says anything about leakage.

Strongest existing audit. The strongest existing audit is the interpolation-resolved membership attack from Where Rectified Flows Leak. It feeds an 11-point reconstruction-error profile along the flow path to a small classifier and reports 0.91 AUC on MAESTRO v3, against 0.72 to 0.83 for diffusion attacks. No reported result was found on Lacuna for this attack, or any membership attack, on LJSpeech or any TTS model. We will run it on every arm using the same member and non-member sentences and the same classifier tuning budget as our audit.

Our canary audit beats it if two things hold. First, it detects memorization at a lower repetition count, defined as the smallest k at which canary exposure is significant at p<0.01. Second, its ranking of arms by leakage agrees better with the natural-data extraction gap. We also report the AUC of the membership attack alone.

Strongest existing mitigation. The strongest mitigation is per-core gradient clipping from Unintended Memorization in Large ASR Models, which reported a WER of 3.87 versus 4.09 without clipping on a 600M speech recognizer. We apply it during teacher training with micro-batch 4 and the same total compute. A second mitigation is adapted from Against Membership Inference Attack: Pruning is All You Need. We prune while penalizing the gap in flow-matching loss between training and held-out sentences. This tests whether privacy-aware pruning removes what plain pruning keeps.

Key ablation. The ablation that isolates the main ingredient is the matched-size dense model trained from scratch. If it shows canary exposure equal to the pruned model's, then lineage does not matter and only size does.

Why not just…?

Why not just run the existing flow membership attack and skip the canaries? Membership scores say whether a sample was seen. They do not say whether its prosody can be pulled back out by typing the sentence. On large audio generators these scores sit near chance even when data inference later succeeds (Membership and Dataset Inference Attacks on Large Audio Generative Models). This is a real threat to our design. On a small single-speaker corpus the interpolation-resolved attack might already be very strong, and canaries would add little. The head-to-head comparison in the baselines settles it. If the attack's arm ranking matches the canary ranking and it is just as sensitive at low repetition counts, we will report the attack as sufficient and narrow our contribution to the compression findings.

Why not just train a small model from scratch and avoid the question? That is exactly the ablation arm. If small-from-scratch models leak as little as we expect, the practical recommendation would be to pay the training cost directly rather than compress. That trade-off needs measuring, because distillation and pruning are popular precisely when a small model trained directly sounds worse. We report MOS and WER for both, so the cost of the safer route is visible.

Why not just add gradient clipping or differential privacy to the teacher? Clipping may well remove canaries in the teacher, and we test that. But Rethinking LLM Memorization through the Lens of Adversarial Compression shows that defenses can hide memorized content from plain prompting while it remains retrievable. We therefore audit clipped teachers after compression and at few steps as well. A mitigation that appears to work in the teacher but leaks again in the 4-bit, 2-step student would be exactly the failure this study is built to catch.

Proposed Experiments and Decisive Checks

All arms are trained on LJSpeech with canaries, plus the LibriTTS low-data arm (LibriTTS). Each arm is reported with WER from an off-the-shelf recognizer and listening-test MOS, so that leakage is always shown next to utility. Leakage is measured three ways: canary exposure rank as a function of repetition count; the member/non-member gap in prosody distance and mel copy similarity; and the AUC of the interpolation-resolved attack. We also record the interpolation point where leakage peaks. This tests whether the few solver steps used by the 2- and 4-step samplers happen to land near it, which would explain any step-count effect.

Lineage check. The decisive check is the comparison at matched non-zero parameter count. If pruned and quantized descendants keep canary exposure statistically indistinguishable from the teacher, while the dense small model drops toward the null distribution, compression-inherits-memorization is supported. The reverse outcome, where the pruned model forgets as much as the dense one, supports the pruning-as-regularizer view and refutes our hypothesis. Either result is useful to anyone shipping on-device voices.

Guidance check. The second check comes from the guided variant (Prior-free Guided TTS), which reported MOS 4.39 at guidance scale 3. If canary exposure rises with guidance scale on the 2-step student, the image-diffusion finding that guidance drives copying carries over to speech, and the quality gain from guidance is partly a copying effect.

Low-data check. The third check is the LibriTTS low-data arm. Here we expect exposure to be highest, given that replication rose sharply at small training-set size in Generation or Replication. We will test whether compression narrows or widens that gap.