A voice carries two things at once: who is speaking, which we call timbre and which comes mostly from the anatomy of the vocal tract, and how they are speaking, which we call prosody or style and which covers pitch movement, rhythm, stress and emotion. Systems that copy one person's speaking style onto another person's voice keep failing in one of two ways. Either the copied style drags the reference speaker's identity along with it, or scrubbing away that identity also flattens the expression. Current fixes rely on very large trained models, adversarial losses and tens of thousands of hours of audio. None of them tells us where in the signal identity actually lives. Older work did answer that question for the pitch contour alone. A wavelet analysis found that identity sits in intermediate timescales, while very fast and very slow pitch movements are shared across speakers (Wavelet Analysis of Speaker Dependent and Independent Prosody).
We propose to answer the same question for the full acoustic signal using reservoir computing. A reservoir is a large recurrent network whose weights are random and never trained; only a linear output layer on top of it is fitted, in closed form. Its useful property here is that one parameter, the leak rate, sets how slowly its state forgets the input. That makes it a tunable, causal, nonlinear filter at a chosen timescale. We drive a bank of such reservoirs, one per timescale, with pitch, energy and spectral features. Linear readouts then measure how much speaker information and how much emotion information each timescale carries. The resulting map is used to build a training-free, streaming style encoder: keep the timescales that carry emotion, and project out the directions that predict speaker. The main scientific question is whether the reservoir's nonlinear mixing of channels finds identity cues that a linear filter bank such as the wavelet transform cannot. The experiments are designed so that a "no" answer is just as informative as a "yes".
Core Claim
Speaker identity in expressive speech is concentrated at specific timescales, and the location depends on the input channel. For pitch we expect it at intermediate scales, as the wavelet study found. For the spectral envelope we expect it at the slowest scales, because long-term averaged spectra are a classic timbre cue. Part of this identity information exists only in interactions between channels, for example how formant positions co-vary with pitch, so per-channel linear filtering cannot reach it. A bank of fixed nonlinear reservoirs at controlled timescales, read out by ridge regression, both measures this structure and removes it. It yields a frame-level style representation with less identity leakage than linear timescale filtering, at a cost of seconds of fitting and no backpropagation.
This differs from the closest work in three ways. The wavelet analysis uses a fixed linear, non-causal basis on one-dimensional pitch and energy contours. We use causal, nonlinear dynamics on high-dimensional spectral input, and we test the linear case directly as an ablation. Disentangling prosody and timbre embeddings via voice conversion trains WavLM-based utterance-level prosody and timbre embeddings using voice-converted data. It reports that prosody alone still verifies speakers at 13.5% equal error rate, which suggests prosody is inherently identity-laden. Our timescale map arbitrates between that finding and the wavelet result. If leakage is scale-dependent, removing specific bands should lower that error rate sharply while keeping emotion information. If it is not, no band selection will help. TVTSyn argues that timbre must be time-varying and models it with learned attention over a timbre memory. We instead obtain time-varying style and identity estimates from deterministic dynamics with closed-form readouts.
Research Program
The core object is a bank of leaky echo state networks. Each network updates its state $x_t \in \mathbb{R}^N$ from frame features $u_t$ as
$$x_t = (1-a)\,x_{t-1} + a\,\tanh(W x_{t-1} + U u_t),$$
where $W$ and $U$ are fixed random recurrent and input matrices. The recurrent matrix $W$ is scaled to a spectral radius below one so that the state is a fading-memory function of past input, the echo state property analyzed by Gallicchio et al.. The leak rate $a\in(0,1]$ sets an effective memory of roughly $1/a$ frames.
We hold the spectral radius, input scaling and sparsity fixed. Only $a$ varies, over eight values chosen so that each reservoir's frequency response for a step input aligns with the wavelet scales of the 2018 study. This alignment lets the two maps be compared index by index. It also sidesteps much of the hyperparameter sensitivity that plagues reservoirs, because we sweep one interpretable knob. As a check, we use the zero-rollout selection idea of Free-Probability Kernels for Zero-Rollout Hyperparameter Selection to confirm that the fixed settings are not degenerate.
The inputs are frame-level log-pitch, energy and 80-band log-mel features, run separately and jointly. Running them separately localizes identity per channel. Running them jointly exposes cross-channel interactions. Every reservoir is instantiated with five random seeds, and all results are averaged over seeds. States are computed once and reused across all readouts and folds, the efficiency argument of Lukosevicius et al..
The probing study uses the parallel structure of the Emotional Speech Dataset, introduced in Seen and Unseen emotional style transfer for voice conversion. Its 10 English speakers each read the same 350 sentences in five emotions, which allows two symmetric tests. For speaker information, we train a speaker readout on some emotions and test on held-out emotions, which measures identity that survives a change of style. For emotion information, we train an emotion readout on some speakers and test on held-out speakers, which measures style that generalizes across voices. Readouts operate on time-pooled states for utterance decisions and on per-frame states for frame-level decoding. VCTK, with its 109 speakers, provides the larger-scale speaker probe, using the same split as the WavLM disentanglement paper. The output is a channel-by-timescale map of both quantities.
The style encoder follows directly from the map. We keep the reservoir bands whose emotion-to-speaker information ratio is highest. We then fit a ridge regression from the retained states to speaker identity and project the states onto the orthogonal complement of its weight vectors. We repeat this fit-and-project step until speaker decoding from the projected states falls to chance on held-out data. The result is a causal, frame-synchronous style trajectory computed with fixed weights plus one projection matrix. This connects to the low-data robustness of reservoirs reported by Hinaut. Because the projection is closed-form, it can be refit for a new speaker population in seconds.
To show that the representation matters for generation, the style trajectory conditions a conventional trained voice-conversion decoder. We do not ask reservoirs to synthesize audio, which the literature treats as unsolved. The decoder is held fixed across all conditions.
Baselines
The naive reference point is the raw frame features themselves: pitch, energy and mel. These are pooled or downsampled to the same dimension as every other style representation and passed through the same readouts and the same decoder. This reference shows how much identity and emotion are linearly available with no temporal processing. A good method must reduce speaker decoding below it without losing emotion decoding.
The most direct competitor on the timescale question is the continuous wavelet decomposition of Wavelet Analysis of Speaker Dependent and Independent Prosody, applied to the same channels with the same scales. On VCC 2016 it reports that converting only the intermediate scales 4 to 8 raised the Pearson correlation from 0.842 to 0.854. In that setting, 82% of listeners identified the target speaker. We will rerun its intermediate-scale protocol on VCC 2016 to confirm that our reimplementation reproduces these figures. We then extend it to the mel channels and to the Emotional Speech Dataset with identical readouts. A causal variant using one-sided exponential filters is included so that causality is not confounded with method.
The ablation that isolates our key ingredient is a linear diagonal reservoir. It has the same leak rates and the same state size, with the tanh removed and $W$ made diagonal. This is the family that ESNv2 and Harnessing Untrained Dynamics use for parallel computation, and it amounts to a random causal filter bank. Following Benchmarking Nonlinear Readouts in Linear Reservoir Networks, we also give this linear bank an explicit quadratic readout over pairwise channel products. This tests whether the cross-channel interactions we attribute to the nonlinear reservoir can be recovered by a cheap polynomial expansion. Beating this ablation is the main claim; beating the raw features is not enough.
The strongest existing learned representation for this question is the W-PRO prosody embedding of Disentangling prosody and timbre embeddings via voice conversion. It reports a speaker-verification equal error rate of 13.5% on converted VoxCeleb data, compared with under 2% for full speaker embeddings, and 65.7% weighted accuracy on IEMOCAP emotion classification. We will reproduce its training recipe on the same converted-VCTK setup and then evaluate it with the same Emotional Speech Dataset probes. The comparison happens on our splits; the paper's figures serve only as a reproduction check.
Beating W-PRO means one of two things on held-out Emotional Speech Dataset speakers. Either our encoder gives a higher speaker equal error rate at equal or better cross-speaker emotion accuracy, or it gives equal leakage at much lower training cost. A trained small GRU of matching output dimension, trained with the same budget and the same speaker-adversary-free objective, rounds out the comparison. This follows the matched-budget logic of Micheli et al..
For the generation experiment, the style encoders are swapped into one shared decoder. This decoder is trained on VCTK plus the Emotional Speech Dataset with the same steps, data and tuning budget per condition, so only the style input differs. Large systems serve as context rather than as matched competitors. StableVC reports a naturalness MOS of 3.96, speaker similarity of 0.67 and word error rate of 2.03% on Emotional Speech Dataset and VCTK evaluation, from roughly 20,000 hours of training. ZSVC reports subjective style similarity of 3.96 after pre-training on 44,000 hours. We do not expect to match these with a small decoder. We will report them so readers can see the size of the gap.
Why not just…?
Why not just use the wavelet transform or any fixed filter bank? This is the real threat. On pitch and energy alone, we expect a wavelet bank to perform about as well as a reservoir, since both are timescale filters. The wavelet bank has two concrete limitations. It is non-causal at slow scales, so it cannot stream. It is also linear and per-channel, so it cannot represent identity cues that live in how pitch and spectral shape move together. The linear-reservoir ablation with and without a quadratic readout settles the question. If nonlinear reservoirs do not beat the linear bank plus quadratic features on the joint pitch-and-mel input, we will conclude that timescale selection, not reservoir dynamics, is what matters. In that case the deliverable becomes a causal filter-bank style encoder plus the identity map, and we will say so.
Why not just train a GRU or a small state-space model? A trained recurrent encoder can in principle learn any of these features. Its timescales, however, are not controlled, so it gives no map of where identity lives. It also needs backpropagation through time and enough data to avoid overfitting, which with only 10 speakers is precisely the regime where Hinaut reports that recurrent networks overfit while reservoirs generalize. The matched GRU baseline tests this directly on held-out speakers. If the GRU wins on leakage at equal emotion accuracy, the reservoir's advantage reduces to interpretability and fitting cost.
Why not just use self-supervised embeddings with a speaker adversary, or discrete codec tokens? W-PRO already obtains strong prosody embeddings from WavLM. Adversarial approaches such as StableVC and ZSVC, and codec bottlenecks such as DC CoMix TTS, all reduce leakage. These approaches still leak and still cost a lot. W-PRO verifies speakers at 13.5% equal error rate. DC CoMix TTS reports that its quantized codes alter word stress. The adversarial systems rely on 20,000 to 44,000 hours of audio. Our encoder does not aim to beat their perceptual quality. It aims to show that a closed-form, causal representation can match or exceed W-PRO's leakage-versus-emotion trade-off on held-out speakers, and to explain that trade-off with a timescale map. The head-to-head probe on identical splits decides whether that holds.
Proposed Experiments
The first decisive check is the timescale map itself. Using pitch and energy input only, our speaker-information curve across leak rates should peak at the timescales aligned to wavelet scales 4 to 8. Agreement there validates the probe against the 2018 result. With mel input, we predict a different shape: speaker information should rise monotonically toward the slowest timescales. If the two curves look the same, our account of channel-dependent identity is wrong.
The second check is the leakage–expression trade-off on held-out Emotional Speech Dataset speakers. We sweep the number of retained bands and the number of projection rounds. For each setting we plot speaker equal error rate against cross-speaker emotion accuracy, and we compare the curves of the raw features, the wavelet bank, the linear reservoir with and without quadratic readout, the nonlinear reservoir, the trained GRU and W-PRO. The proposal succeeds if the nonlinear reservoir's curve lies strictly above the linear ablations at intermediate and slow timescales on the joint input, with confidence intervals taken over the five reservoir seeds. It fails if the linear bank with quadratic readout matches it.
The third check addresses the tension in the literature. On the converted-VCTK protocol, we measure W-PRO's equal error rate before and after the same band removal and projection are applied to time-resolved WavLM features. If leakage is scale-dependent, the error rate should rise well above 13.5% while IEMOCAP emotion accuracy stays near 65.7%.
The final check is generation. Using the shared decoder, we convert Emotional Speech Dataset utterances to unseen target voices and measure four things: speaker similarity to the target as cosine similarity of speaker embeddings, emotion reclassification accuracy (the metric used by ED-TTS), pitch-contour correlation with the source, and word error rate. Small listening tests rate naturalness and style similarity. We also record encoder latency on CPU against the roughly 132 ms total CPU latency reported for TVTSyn. The encoder is causal and requires only matrix multiplies, so its own latency should be negligible next to the decoder's. If the generation results do not follow the probe ordering, the probe has measured something the decoder cannot use, and that would be a substantive negative result.
References
- Wavelet Analysis of Speaker Dependent and Independent Prosody for Voice Conversion
- Disentangling prosody and timbre embeddings via voice conversion
- TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymization
- Randomized Machine Learning Approaches: Recent Developments and Challenges
- Free-Probability Kernels for Zero-Rollout Hyperparameter Selection in Reservoir Computing
- Efficient Implementations of Echo State Network Cross-Validation
- Seen and Unseen Emotional Style Transfer for Voice Conversion with A New Emotional Speech Dataset
- Robot Language Acquisition Modelling via Cross-Situational Learning with Little Data
- ESNv2: Resurrecting Reservoir Computing in the Deep Learning era
- Harnessing Untrained Dynamics: A Reservoir Computing Approach to State-Space Models
- Benchmarking Nonlinear Readouts in Linear Reservoir Networks
- Comparison between DeepESNs and gated RNNs on multivariate time-series prediction
- StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow Matching
- ZSVC: Zero-shot Style Voice Conversion with Disentangled Latent Diffusion Models and Adversarial Training
- DC CoMix TTS: An End-to-End Expressive TTS with Discrete Code Collaborated with Mixer
- ED-TTS: Multi-Scale Emotion Modeling Using Cross-Domain Emotion Diarization for Emotional Speech Synthesis