When an expressive text-to-speech (TTS) model is trained on data that cannot be pooled, for example recordings held by different studios, each specializing in certain emotions, speakers, or speaking styles, the natural way to train without a central server is gossip averaging. In gossip averaging, each machine trains on its own data and repeatedly averages its weights with a few neighbors. We conjecture that this setting has a failure mode that standard training dashboards will not catch. The disagreement between machines' models, which optimization theory calls consensus error, should concentrate in the parts of the network that encode style. The visible symptom should be prosodic flattening: synthesized speech keeps a low word error rate but loses pitch range and the contrast between emotions. This is the speech community's familiar "regression to the mean" (ESPnet-TTS), produced this time by the communication protocol rather than by the loss function.
If the conjecture holds, it implies a cheap fix. Full gradient tracking corrects disagreement caused by uneven data across machines, but it roughly doubles per-node state and traffic. We would apply it only to the parameter blocks where consensus error is measured to concentrate, and send heavily compressed gossip everywhere else. The note proposes to test whether the flattening is real and localized. It then tests whether selective tracking recovers the expressiveness of centralized training at a fraction of the communication cost of tracking the whole model.
Core Claim
Our claim has two parts. Under style-partitioned data, plain decentralized SGD degrades expressive range (per-style pitch variance, emotion separability, style-prompt adherence) much earlier and much more than it degrades intelligibility. Consensus error is also unevenly spread across the network. It sits mostly in style-conditioning blocks: the prompt encoder, the cross-attention into the prompt, and the duration and prosody heads. For each parameter block $b$ we track the per-block consensus distance
$$\Xi_b(t) = \frac{1}{n}\sum_{i=1}^{n} \lVert \theta_{i,b}(t) - \bar{\theta}_b(t) \rVert^2 ,$$
where $n$ is the number of machines, $\theta_{i,b}(t)$ is machine $i$'s copy of block $b$ at step $t$, and $\bar{\theta}_b(t)$ is the average across machines. The method is block-selective tracking. Blocks whose $\Xi_b$, normalized by block size, ranks in the top fraction get gradient tracking in the push-pull style. All other blocks use error-compensated compressed gossip. The ranking is recomputed periodically. The prediction is that this matches full tracking on style metrics at close to the communication cost of compressed plain gossip.
Relation to Prior Work
Two papers motivate the gap. Asynchronous Decentralized Distributed Training of Acoustic Models trains speech-recognition LSTM-HMM classifiers with ring and random-partner gossip on 128 GPUs. It reaches 7.7% Switchboard WER versus 7.5% on a single GPU, and it is resilient to stragglers. It covers a discriminative frame classifier on effectively shuffled data, so it says nothing about generative, style-conditioned models trained on heterogeneous shards, which is our setting. UniFLoW shows that federated low-rank adapters for multimodal models need careful aggregation. It averages one LoRA factor and solves for the other on a server. It addresses audio understanding rather than generation, and it keeps a central coordinator, which we remove.
Three papers supply the method and constrain it. CHOCO-SGD supplies error-compensated compressed gossip, which reaches uncompressed accuracy with 100x fewer bits on epsilon and rcv1. We use it for the low-priority blocks. B-ary Tree Push-Pull supplies gradient tracking over bounded-degree trees. It tracks centralized SGD on heterogeneous MNIST and CIFAR-10, where plain decentralized SGD fluctuates or fails. We use its tracking on the high-priority blocks only. Enhancing Parallelism in Decentralized Stochastic Convex Optimization shows that plain decentralized SGD degrades on rings with as few as 4 machines. That result constrains the design: plain gossip without a fix is not a viable baseline beyond a few nodes. The same paper's remedy also doubles communication. That cost, shared by tracking methods, is exactly what selective application tries to avoid.
On the speech side, Sparse Autoencoders for Interpretable Emotion Control in TTS shows that emotions are carried by sparse, localized features in a TTS backbone's residual stream. This is indirect support for our localization hypothesis, since style information really does live in identifiable places. SPAM provides an automatic prompt-adherence metric that correlates with human scores, and we need such a metric to detect flattening cheaply during training. We do not build on Revisiting Parameter Server in LLM Post-Training. Its fix targets layer-wise synchronization bubbles from variable sequence length, and for sentence-level TTS, length bucketing already handles that problem reasonably well.
Research Program
We would train a prompt-conditioned codec language model in the hierarchical design of Salle from TextrolSpeech, at roughly 0.2B parameters so that runs at 16 nodes are affordable. One stage autoregressively predicts first-level codec tokens from text and a style prompt, and a second stage fills in the remaining codebooks. The base pool is ParaSpeechCaps (Scaling Rich Style-Prompted TTS Datasets), with 2,700 hours and 59 style tags. TextrolSpeech and LibriTTS-P serve for evaluation and a second partition study.
We would build three partitions across 8 and 16 nodes. The first is IID shuffled. The second is style-partitioned, with each node holding a disjoint subset of style tags. The third is speaker-partitioned, which separates heterogeneity in timbre from heterogeneity in prosody. Topologies are ring, torus, and B-ary tree, with link bandwidth emulated so that total bits are a controlled variable.
The analysis has three steps. First, we log $\Xi_b$ for every block throughout training under plain decentralized SGD, alongside pitch standard deviation per style, emotion separability, SPAM adherence, and WER on a fixed probe set. This tests whether flattening appears and whether it co-moves with consensus error in style blocks. Second, we implement block-selective tracking with a fraction $k$ of tracked blocks, sweeping $k$ from 0 to 1, and report expressiveness against bits transmitted. Third, we check whether the selected blocks are stable across seeds and topologies. If they are, a fixed, architecture-derived selection would work without online measurement.
Baselines
The upper reference is centralized synchronous training with all-reduce on IID-shuffled data, using the same model, total samples, optimizer, and tuning budget. It shows what expressiveness is achievable when there is neither heterogeneity nor consensus error. For context, the original TextrolSpeech paper reports a mean style-matching accuracy of 87.6% for Salle trained centrally (versus 82.3% for PromptTTS), and SPAM reports a correlation of 0.584 with human MOS on TextrolSpeech. Our centralized run should land in that neighborhood before the decentralized comparisons mean anything.
The naive reference points are local-only training, where each node trains on its own shard with no communication, and plain decentralized SGD without compression or tracking. The strongest existing decentralized competitor is full gradient tracking with push-pull over B-ary trees as in BTPP, applied to all parameters. Since it is designed specifically for heterogeneous data, it is the method we must match on style metrics. The strongest bandwidth competitor is CHOCO-SGD at 1% top-k sparsification on all blocks. A centralized star with low-rank adapters, aggregated in the manner of UniFLoW, represents the server-based alternative. Every method gets the same number of samples, the same learning-rate sweep (five values), and is compared at equal bits transmitted per node. There is no reported result found on Lacuna for decentralized or federated training of any TTS model, so the decentralized numbers will be new.
The key ablation isolates the choice of blocks. We track a randomly chosen set of blocks with the same parameter count, and separately the blocks with the lowest $\Xi_b$. Beating the competition means matching full BTPP within one point of style-matching accuracy and SPAM adherence at no more than 60% of its per-node traffic, while beating CHOCO-only at equal bits. Random-block tracking at equal budget must also clearly underperform. Otherwise the gain comes from tracking per se, not from localization.
Why not just…?
Why not just shuffle the data so every node is IID? In a single datacenter, one should. Our regime is data that cannot move: studio-owned or consented recordings, or geo-distributed sites with limited links. In that regime shuffling is not an option. The IID partition is still included as a control. If flattening appears under IID partitions as well, the cause is consensus error on sparse topologies, not heterogeneity, and the claim must be revised.
Why not just use full gradient tracking and accept the cost? This is the main threat. If BTPP's overhead turns out negligible at our model size and bandwidth, selective tracking adds complexity for little benefit. The bits-versus-expressiveness curve settles this directly. If full tracking dominates at every bandwidth we test, we report the localization finding as diagnostic only.
Why not just train a neutral model centrally and steer emotions afterward? Post-hoc steering needs the style features to exist. The SAE emotion-control work finds them in a backbone already exposed to emotional speech. Crude global steering raised WER to 2.86% versus 0.57% for sparse-feature steering, which shows the cost of steering without good features. Since the expressive data is exactly what sits on separate sites, steering does not remove the need to train on it.
Why not just keep a central server with adapters? For a handful of sites this is reasonable, and it is a baseline. Its weaknesses are the hub's bandwidth growing linearly with the number of clients (FireCaffe) and a single point of failure. We test it at 8 and 16 nodes. If it matches decentralized methods at equal bits, the honest conclusion is that topology matters less than tracking.
Decisive Checks
The proposal stands or falls on four observations, all made on the TextrolSpeech and ParaSpeechCaps test sets. The first is whether plain gossip under style partitioning lowers per-style pitch variance and SPAM adherence while leaving WER near the centralized value. The second is whether $\Xi_b$ in style-conditioning blocks exceeds that of other blocks after normalizing by block size, consistently across seeds and topologies. The third is whether block-selective tracking reaches BTPP's style-matching accuracy at substantially lower traffic. The fourth is whether random-block tracking at equal budget fails to do so. A small human MOS and instruction-consistency study on the top three systems confirms that the automatic metrics track perceived expressiveness. If the first check fails, the note's hypothesis is wrong, and that negative result is itself worth reporting to anyone training speech models across sites.
References
- Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit
- Asynchronous Decentralized Distributed Training of Acoustic Models
- UniFLoW: Universal Multi-Modal Federated LoRA Fine-Tuning Framework with Analytical Aggregation
- Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
- B-ary Tree Push-Pull Method is Provably Efficient for Distributed Learning on Heterogeneous Data
- Enhancing Parallelism in Decentralized Stochastic Convex Optimization
- Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech
- SPAM: Style Prompt Adherence Metric for Prompt-based TTS
- Revisiting Parameter Server in LLM Post-Training
- TextrolSpeech: A Text Style Control Speech Corpus with Codec Language Text-to-Speech Models
- Scaling Rich Style-Prompted Text-to-Speech Datasets
- LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioning
- FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters