Echo State Networks with Learned Neuron Gains for Telephone Speech Bandwidth Extension

Telephone channels cut speech off at about 3.4 kHz. Bandwidth extension tries to guess the missing 4–8 kHz band from the part that survives, and the best current systems do this with deep convolutional networks. These have millions of weights, are trained for many epochs by backpropagation, and have to be retrained whenever a design choice changes. Reservoir computing offers a much cheaper recipe. It feeds the signal through a large recurrent network whose weights are random and never trained (the "reservoir"), then fits only a linear output layer, which can be done in closed form with ridge regression. This recipe has never been tried on bandwidth extension. The one close speech result is a warning: it shows that untrained recurrent layers fail when asked to process raw acoustics.

We propose to test whether a small, targeted exception to "never train the reservoir" closes that gap. The recurrent weights stay random and fixed. We learn only a few thousand internal numbers: one output amplitude per neuron, plus the global input scaling, recurrence scaling and leak rate. These are learned by gradient descent on a validation loss, and the linear readout is re-solved exactly at every step. The hypothesis is specific to speech. A fixed reservoir saturates on loud vowels and goes nearly silent on quiet fricatives such as /s/ and /f/, and the fricatives are exactly where most high-band energy lives. Per-neuron amplitudes let different neurons specialise to different loudness ranges. If this is right, the gain should show up specifically on fricatives, and a single global scaling or plain input normalisation should not reproduce it.

Core Claim

A fixed-weight Echo State Network (ESN) can serve as the spectral-envelope estimator in a classical bandwidth extension pipeline. An ESN is a reservoir network whose random recurrent weights are scaled so that its state fades smoothly with its input history. The other half of the pipeline, the harmonic fine structure, is regenerated by spectral folding, which mirrors the low-band excitation into the high band. Two further claims are testable. First, neuron-wise learned activation amplitudes, optimised through a closed-form readout, reduce high-band log-spectral distortion beyond what the best matched-budget tuning of global reservoir hyperparameters achieves. Second, most of that reduction is concentrated on low-energy fricative frames. The practical target is a point on the quality-versus-cost curve, not a new quality record. We want wideband quality within reach of a trained temporal convolutional network, at a fraction of its training cost and per-frame compute, with the recurrent weights recoverable from a random seed.

The division of labour is grounded in perceptual evidence. The effect of highband harmonic structure in the artificial bandwidth expansion of telephone speech compared plain spectral folding against high-band harmonics locked to the estimated pitch. Its listening tests found no significant preference ($p = 0.37$). That result tells us not to spend model capacity on exact harmonic placement and to put it into the envelope instead. The two-stage view in A Two-Stage Approach to Speech Bandwidth Extension partly contradicts this. That paper predicts a high-band spectrogram with a temporal convolutional network and then adds a trained Wave-U-Net stage to polish the waveform, arguing the second stage is needed. We take that disagreement as an empirical question that our comparison directly answers. We do not assume folding is sufficient.

Research Program

The system follows the source-filter decomposition used by Mapping neural networks for bandwidth extension of narrowband speech. That paper maps narrowband cepstra to wideband cepstra with a static four-layer feedforward network, speaker by speaker, and upsamples the linear-prediction residual by zero insertion to fold harmonics upward. We keep the folding and the all-pole synthesis. We replace the static per-frame network with a reservoir that carries memory across frames, and we train and test across speakers.

The reservoir's inputs are narrowband log-spectral or cepstral features plus frame log-energy. Its readout predicts the high-band envelope. The state update with the extended parameters is

$$x_t = (1-a)\,x_{t-1} + a\,\lambda \odot \tanh\!\big(\rho\, W x_{t-1} + s\, W_{in} u_t\big),$$

where $x_t$ is the $N$-dimensional reservoir state at frame $t$ and $u_t$ is the narrowband feature vector. $W$ and $W_{in}$ are fixed sparse random matrices generated from a seed. The global parameters are $\rho$ (recurrence scaling), $s$ (input scaling) and $a$ (leak rate). $\lambda \in \mathbb{R}^N$ is the vector of per-neuron amplitudes, and $\odot$ is elementwise product.

Writing $\theta = (\rho, s, a, \lambda)$, the readout is the exact ridge solution on the training split, $W_{out}(\theta) = Y X_\theta^\top (X_\theta X_\theta^\top + \beta I)^{-1}$. Here $X_\theta$ stacks the training states and $Y$ the target envelopes. The parameters $\theta$ are updated by gradient descent on the validation error of $W_{out}(\theta)$. The gradient passes through the linear solve and through a truncated unrolling of the state update. Because $\theta$ has only about $N+3$ entries and the solve is differentiable, this costs orders of magnitude less than training a convolutional network. The readout is never trained by iterative descent.

This design draws on three sources, each in a different role. Echo State Speech Recognition supplies the idea of learning reservoir and input scaling by gradient while keeping recurrent weights fixed and seed-generated. It trains those scalings jointly with a full recognition network and uses reservoirs only in the decoder. We instead solve the readout exactly, use the reservoir as the acoustic-mapping stage itself, and add neuron-wise terms. Recurrent neural networks with trainable amplitude of activation functions supplies the neuron-wise amplitude. It learns amplitudes by real-time recurrent learning in fully trained networks and lowers mean squared error on synthetic AR(4) coloured noise from 0.0309 to 0.0116. We move that parameter into a fixed reservoir and a real acoustic task. Finally, Unsupervised learning of echo state networks: balancing the double pole constrains the design. It found that evolutionary tuning of $\rho$ helped a control task but gave the worst result on a supervised sine-prediction task. We treat that as a live risk rather than dismissing it. Every internal parameter is fit on validation loss, never training loss, and we report whether tuning $\rho$ helps or hurts on its own.

Beyond quality numbers, the analysis will characterise what the amplitudes do. It will measure the fraction of saturated units (|pre-activation| beyond the tanh knee) on voiced versus fricative frames, and compare the learned $\lambda$ distribution against units' average input energy. It will also check whether the solution survives changing the random seed.

Baselines

The naive reference point is narrowband speech upsampled with no high band. Right after it comes plain spectral folding with a fixed smoothing envelope, as in Quality improvement of telephone speech by artificial bandwidth expansion - listening tests in three languages. Any learned envelope must clearly beat folding. Artificial Bandwidth Extension Using H∞ Optimization reports a comparative mean opinion score gain of 0.76 for a GMM-based approach over spectrum folding. That is the scale of improvement a learned envelope is known to buy.

The strongest existing method on quality is the TCN with spectral-refinement stage (TCN+SS) of A Two-Stage Approach to Speech Bandwidth Extension. With about 2.7M parameters it reports WB-PESQ 4.08 on the Valentini-Botinhao corpus. After extension it also reports LibriSpeech word error rate 12.67%, against 18.27% for a Wave-U-Net baseline and 11.19% for the original wideband audio. We will retrain it on our exact training data with its published loss. We will also train its stage-1 TCN alone with mirrored phase, which isolates the waveform-refinement question. For deployment cost, the reference is Streaming SEANet from Real-time Speech Frequency Bandwidth Extension, which reports 1.5 ms compute per 16 ms frame on one Pixel 4 CPU core. We will compare multiply-accumulates per frame and single-core latency on the same hardware class. Beating TCN+SS outright is not required. The proposal succeeds if the reservoir system stays within about 0.2 WB-PESQ and one to two points of downstream WER of it, while needing at least ten times less training compute and fewer per-frame operations. If the gap is larger than that at every reservoir size we try, the claim fails.

Two ablations isolate the key ingredient. The first is a fixed ESN whose global hyperparameters ($\rho$, $s$, $a$, $\beta$) are chosen by grid or Bayesian search over cached state matrices, using the fast cross-validation of Efficient Implementations of Echo State Network Cross-Validation. It gets the same wall-clock budget as our gradient procedure. The second is our gradient procedure restricted to the three global parameters. Beating the first shows the learning procedure matters. Beating the second shows the neuron-wise amplitudes matter. We will count the amplitudes as a real contribution only if they lower fricative-frame high-band distortion beyond both ablations across five random seeds, with the difference exceeding seed-to-seed spread.

Why not just…?

Why not just tune a fixed reservoir's global hyperparameters by search, since states can be cached and cross-validation is nearly free? This is the most serious threat. Reservoir computing's own tooling makes such search cheap, and three or four global knobs may capture most of what matters. Search cannot reach an $N$-dimensional amplitude vector, but that does not show the vector is needed. The matched-budget search ablation above settles it directly. If searched global parameters match the full model on fricatives, we will report that the extended parameters buy nothing here, and the deliverable becomes a well-tuned fixed-reservoir bandwidth extender.

Why not just normalise input loudness per frame instead of learning neuron gains? Automatic gain control or per-frame energy normalisation, with log-energy fed as an extra input, is a simpler fix for the dynamic-range problem. Artificial bandwidth extension without side information for ITU-t g.729.1 attributes its muffled high band (about 5.05 dB RMS high-band distortion against 3.55 dB for the true coded band) to limited dynamic range. That suggests the difficulty is real, but it does not show that input normalisation cannot solve it. We will run a fixed ESN with normalised inputs and an energy channel as a baseline. Normalisation removes absolute level, which carries information about the high band. Per-neuron gains could in principle keep level information while avoiding saturation. Whether that matters is exactly what the per-phoneme comparison measures.

Why not just use a small trained recurrent or convolutional network at matched parameter count? A small GRU or a shrunken TCN trained by backpropagation is the obvious competitor at low cost. A Two-Stage Approach to Speech Bandwidth Extension shows a TCN matching a convolutional recurrent network with 84% fewer parameters. We will train a GRU and a TCN at the same trained-parameter count as our readout plus $\theta$, and a second pair at the same per-frame compute, each with a tuning budget equal to ours. If they win on both quality and training cost, reservoirs have no niche here. We regard this as a plausible outcome and will report it.

Why not just a static feedforward map over stacked context frames? Mapping neural networks for bandwidth extension of narrowband speech uses a static feedforward network, and one could simply stack neighbouring frames to supply context. We include ridge regression on stacked frames, and random nonlinear features over stacked frames without recurrence, at equal feature dimension. This isolates whether recurrent memory contributes anything over a fixed context window.

Proposed Experiments

Training uses TIMIT with gender- and dialect-balanced speakers. Testing uses the 400-file TIMIT test protocol of Artificial Bandwidth Extension Using H∞ Optimization. Narrowband input is produced both by that paper's direct downsampling and by a 300–3400 Hz telephone filter with AMR-NB 12.2 kbit/s coding. TIMIT's phone alignments let us split high-band log-spectral distortion by phone class: fricatives, voiced vowels, stops and silence. The central prediction is that the amplitudes help fricatives most. If the gains are uniform across classes, or concentrated on vowels, the dynamic-range explanation is wrong even if average quality improves.

Speaker-independent generalisation is tested on GRID in the speaker-dependent and speaker-independent splits of On representation learning for artificial bandwidth extension. There, general stochastic networks, sum-product networks and contractive autoencoders are reported above 4.36 WB-PESQ, with general stochastic networks gaining 3.90 dB (speaker-dependent) and 4.08 dB (speaker-independent) segmental SNR over sum-product networks. No absolute segmental SNR is reported on Lacuna for those models, so we will compare against our own reimplementations.

WB-PESQ on Valentini-Botinhao and downstream LibriSpeech WER provide the direct comparison with TCN+SS. The evaluation of An Objective Evaluation Methodology for Blind Bandwidth Extension shows that a system can raise perceptual scores by quietly suppressing high-band energy. To guard against this, we will run POLQA on ITU-T P.501 material transcoded with AMR 12.2 and check compliance with the 3GPP wideband receive mask. Any system that fails the mask is reported as such, whatever its PESQ. A MUSHRA listening test with folding, TCN+SS, the searched fixed ESN and the full model will decide the question of whether the waveform-refinement stage is needed that the Pulakka and Lin papers disagree on. Cost is reported as training wall-clock, per-frame multiply-accumulates and single-core latency, with recurrent weights stored as a seed.

The decisive checks are three. First, the full model beats the matched-budget searched ESN and the normalised-input ESN on fricative high-band distortion across seeds. Second, the fraction of saturated units on voiced frames drops while fricative accuracy rises. Third, the resulting system sits on the quality-cost curve where TCN+SS and small trained networks do not.