Speaker-Set Memory Time Constants in Bidirectional Linear Recurrence for Multi-Speaker Speech Synthesis

Speakers differ in what they sound like, but they also differ in how their voices move over time. Some let pitch fall slowly across a phrase while others reset it often. Some start shaping the next sound early, and others carry the last sound over. Multi-speaker text-to-speech systems usually represent a speaker as a single vector added to the network's input. That works for timbre, the static colour of a voice. It gives the network no direct handle on the time scales at which a speaker's pitch, energy and spectrum change. We propose to build that handle into the sequence model itself.

The model is a bidirectional linear recurrent network: two simple recurrences, one reading the utterance forward and one reading it backward, each with a learned "forgetting rate" per memory channel. In our version, the speaker's embedding sets those forgetting rates, and it sets them separately for each direction. Speaker identity then becomes, in part, a set of memory time constants. Adapting to a new voice means retuning how long the model remembers what came before and how far it anticipates what comes next.

Core Claim

A single-speaker acoustic model learns one speaker's temporal habits as fixed parameters of its sequence model. This is one concrete reading of the observation that single-speaker training turns personal idiosyncrasies into supposedly universal speech rules (Honnet et al.).

Linear recurrent units (LRUs) make this concrete. An LRU is a recurrent layer whose state update is linear and diagonal, so it trains with a parallel scan instead of step-by-step loops (BLUR). In an LRU, those habits sit in the per-channel decay rates. We let them vary with the speaker:

$$h^{f}_t = \lambda^{f}(s)\odot h^{f}_{t-1} + B^{f}x_t,\qquad h^{b}_t = \lambda^{b}(s)\odot h^{b}_{t+1} + B^{b}x_t,\qquad \lambda^{d}(s)=\exp\!\big(-\exp(\nu^{d}+W^{d}s)\big).$$

The symbols are as follows. $x_t$ is the frame-level linguistic input after duration upsampling. $h^{f}_t$ and $h^{b}_t$ are the forward and backward states. $s$ is the speaker embedding, and $d\in\{f,b\}$ indexes direction. $\nu^{d}$ is a speaker-independent base log-rate. $W^{d}$ is a low-rank map from speaker to rates. The double exponential keeps every rate in $(0,1)$, which preserves the stability guarantee BLUR relies on. Each channel's time constant is $-1/\log\lambda$, so a speaker moves channels between short coarticulatory memory and long phrase-level memory.

The claim we will test has three parts. First, speaker-set time constants improve prosodic and speaker match for unseen speakers in few-shot adaptation compared with input-appended speaker vectors. Second, the learned forward and backward rates track measurable speaker traits, namely pitch autocorrelation time and declination slope. Third, the forward/backward asymmetry varies across speakers more than chance.

Relation to Prior Work

Three older systems motivate the design and set its contrast.

Fan et al. showed that a bidirectional LSTM (a gated recurrent network run in both directions) smooths acoustic trajectories without hand-made delta features. It did so on one speaker, with a log spectral distance of 3.54 dB against 3.73 dB for a feedforward network.

Speaker Representations for Speaker Adaptation in Multiple Speakers' BLSTM-RNN-Based Speech Synthesis took the step closest to ours. It appended i-vectors or speaker codes to a bidirectional LSTM's input, trained on four CMU ARCTIC speakers, and adapted to three more. Its speaker information enters only as an input offset. It never asks whether speakers differ in temporal dynamics.

SRM2TTS separated a global speaker embedding from a BLSTM prosody encoder over six speakers. Its explicit pitch and duration controls degrade beyond roughly 20% and 50% adjustment. This suggests that rescaling levels is not the right knob for speaker-level dynamics.

The method comes from two recent architectures. BLUR supplies the bidirectional LRU with non-linear fusion layers between scans. DeltaFlow supplies the precedent of conditioning recurrent memory gates on an external variable, in its case the diffusion noise level. We swap the noise level for speaker identity. Neither paper touches speech.

ELF constrains our claim. It shows that one static speaker vector causes content-style mismatch, and it fixes that by attending over a speaker codebook conditioned on phonemes. ELF's correction varies speaker traits across phonetic contents. Ours varies them across time scales. The two are complementary, and we compare against ELF directly.

Research Program

We build a non-autoregressive acoustic model with these parts:

  • a phoneme encoder;
  • external durations from forced alignment, following the decoupling argument of Klimkov et al.;
  • a FastSpeech 2–style variance adaptor;
  • a decoder of stacked bidirectional LRU blocks with GLU fusion.

The decoder predicts mel spectrograms, which BigVGAN turns into waveforms. The only thing varied across our main variants is where the speaker embedding enters the decoder. It can enter as an input append, as FiLM scaling of activations, as an input to frame-dependent gates, or as the speaker-only decay map above.

Training uses LibriTTS and VCTK, holding out VCTK speakers for adaptation. A single-speaker LJSpeech model serves as a diagnostic.

The analysis has three parts:

  1. We extract per-speaker time constants from the learned $\lambda(s)$. We correlate them with signal statistics computed directly from each speaker's recordings: the time for F0's autocorrelation to fall off, phrase-level declination slope, and spectral flux.
  2. We test whether forward and backward rates separate along speaker lines.
  3. We run a "how much of a speaker is dynamics" experiment. We take the LJSpeech-trained decoder and adapt it to new speakers by updating only the decay rates, only an input embedding, or both. We measure how much of the speaker-similarity gap each closes.

Baselines

The naive reference is a speaker-independent version of the same LRU decoder with no speaker input. It mirrors the SI system of Speaker Representations…. A second weak reference is the single-speaker LJSpeech decoder used unchanged on other voices.

The classic architectural baseline is a bidirectional LSTM decoder with speaker code appended at the input, as in that 2016 paper. It matches the LRU decoder in parameter count and uses the same encoder, durations and vocoder. The file gives no numeric MCD values for that paper, only that it beat speaker-independent and scratch-trained models, so we reproduce it ourselves.

The strongest attention-based baseline is a FastSpeech 2–style Transformer decoder with a learned speaker embedding. It shares everything except the decoder. Its speaker-free counterpart on LJSpeech scores about 3.85 MOS in DailyTalk's report of the LJSpeech reference.

The strongest zero-shot method is ELF. It reports an unseen-speaker similarity MOS of 3.85 on VCTK, against 3.73 for a trained VITS, and a MOS of 3.98 against 3.95 for ground truth. Those are ELF's own listening-test numbers and are not comparable to new tests. We will therefore run ELF's released configuration on our held-out VCTK speakers in the same listening sessions.

The key ablation compares input-appended and FiLM conditioning on the identical bidirectional LRU. It also includes a variant where the speaker feeds frame-dependent gates, as in a Gated Delta Network, which isolates whether the speaker-only time-constant structure matters. A further ablation ties forward and backward rates, to test whether direction-specific rates matter.

Fairness rules apply to every system:

  • same training data and durations;
  • same vocoder;
  • parameter counts within 5%;
  • the same number of training steps;
  • the same small hyperparameter search of about twelve configurations, chosen on a validation speaker set.

"Beating" a baseline means better speaker similarity and F0 correlation on held-out speakers at 5 and 20 adaptation utterances, with paired-test significance across speakers and no loss in MOS or ASR word error rate.

Why not just…?

Why not just use a Transformer decoder with a speaker embedding? At sentence length, attention cost is not the bottleneck, so efficiency is not our argument. The arguments are representational and parameter-efficiency ones. A speaker embedding added to attention inputs has no explicit knob for memory duration, and adapting a Transformer to a new voice usually means touching many weights. If the matched FastSpeech 2–style baseline equals our model on held-out speaker similarity and dynamics statistics at every adaptation size, the claim fails. The head-to-head comparison is the test.

Why not just append the speaker to the input of a gated recurrence, whose gates then become speaker-dependent anyway? This is the most serious threat. An LSTM or Gated Delta Network can in principle learn speaker-dependent forgetting from an input-appended code. Our ablation with frame-dependent gates fed the speaker settles it directly. If that variant matches the speaker-only decay map, the contribution shrinks to interpretability and to cheap adaptation of a small set of rate parameters, and we will report it that way. We expect the explicit version to win mainly in the few-shot regime, where a low-dimensional, utterance-constant control is easier to estimate from five utterances than an implicit, frame-varying one.

Why not just predict per-speaker pitch, energy and duration explicitly? The variance adaptor is present in every system, so speaking rate and pitch level are already handled identically everywhere. What explicit scalars do not capture is how quickly contours relax and how far they anticipate. Scaling those scalars hard degrades naturalness, as SRM2TTS reports. The dynamics-statistics evaluation below measures exactly what the adaptor misses.

Why not just use ELF's codebook conditioning? ELF models how a speaker's traits change with phonetic content, while ours models time scale. A model could need both. We run ELF as a baseline and also a variant that combines its codebook with speaker-set rates. If the combination adds nothing over ELF alone, the time-constant mechanism is redundant with content-dependent conditioning.

Proposed Experiments

The first decisive check is few-shot adaptation on held-out VCTK speakers with 1, 5 and 20 utterances. We report the following for every system:

  • mel-cepstral distortion;
  • F0 RMSE and F0 correlation;
  • speaker-verification cosine similarity;
  • ASR word error rate;
  • MOS and similarity MOS, collected in a single shared listening test.

We add a dynamics metric: the absolute error between synthesized and real speech in each speaker's F0 autocorrelation time and declination slope. This is where the hypothesis predicts the largest gain.

The second check is interpretive. Learned per-speaker time constants should correlate with those same measured statistics across the roughly hundred VCTK speakers and the LibriTTS speakers. A null correlation would mean the rates are not doing what we claim, even if scores improve.

The third check uses Expresso-style data, where four speakers read identical text in several styles. We test whether style shifts the inferred time constants within a speaker as much as identity shifts them across speakers. If style dominates, the "speaker" rates are really style rates, which matters for the timbre-prosody disentanglement problem. It would also call for separate style and speaker rate maps.

The fourth check is the LJSpeech diagnostic: what share of the similarity gap to a new speaker is closed by retuning only the decay rates. It quantifies how much of the single-speaker limitation sits in learned dynamics rather than timbre.

The fifth check concerns forgetting. Sequential adaptation that updates only per-speaker rate vectors cannot overwrite earlier speakers, so we compare its retained speaker accuracy with the experience-replay results of Hemati & Borth, who found weight regularization collapsed to the newest voice. We treat this as a side benefit, not the core claim.