Traditional digital signal processing relies heavily on tools like the Short-Time Fourier Transform (STFT). These tools operate under a convenient assumption: if you zoom in close enough—typically to a window of 20 to 30 milliseconds—a speech signal becomes "locally stationary." Within this brief window, we assume its statistical properties, such as mean, variance, and frequency content, remain constant.
However, human speech is fundamentally non-stationary. Forcing a fluid, rapidly changing acoustic signal into rigid, fixed-duration blocks introduces a fundamental mismatch between our mathematical models and physical reality.
The Illusion of Local Stationarity
The limitations of assuming stationarity were recognized early in the development of speech processing. As Glen J. Culler noted in An Attack on the Problems of Speech Analysis and Synthesis with the Power of an On-Line System, speech is not composed of infinite periodic sine waves; it consists of brief, localized energy bursts and transient wave packets.
When we enforce a fixed-window analysis, we blur the boundaries of rapid acoustic events. As argued by Daniel Rudoy et al. in Time-varying autoregressive tests for multiscale speech analysis, the spectral characteristics of speech continuously evolve due to the physical dynamics of the vocal tract. Forcing these fluid transitions into fixed blocks limits our ability to capture subtle vocal events. For instance, Abdolreza Sabzi Shahrebabaki et al. point out in Raw Speech-to-Articulatory Inversion by Temporal Filtering and Decimation that transient speech sounds, such as plosives, exhibit massive acoustic shifts within a standard analysis window, causing traditional feature extraction methods to lose vital dynamic information.
Continuous and Time-Varying Parametric Modeling
To capture sub-window dynamics, researchers shifted from piecewise-stationary approximations to models where parameters vary continuously over time.
Traditional linear prediction analysis calculates a static set of filter coefficients for an entire frame. In contrast, time-varying linear prediction (TVLP) and time-varying autoregressive (TVAR) models treat filter coefficients as dynamic trajectories. Karl Schnell et al. demonstrated in Time-varying pre-emphasis and inverse filtering of speech that modeling the continuous evolution of the vocal tract eliminates the block-wise artifacts and inaccuracies that plague traditional speech analysis and synthesis.
Similarly, Paavo Alku developed a method in Time-Varying Quasi-Closed-Phase Weighted Linear Prediction Analysis of Speech for Accurate Formant Detection and Tracking that applies time-continuity constraints to track rapid vocal tract changes even within a single glottal cycle. By moving away from static segment assumptions, these approaches allow speech synthesis and formant tracking to mirror the physical reality of human articulation.
This philosophy extends to sequence modeling as well. Traditional Hidden Markov Models (HMMs) assume that speech segments within a given state are stationary. To overcome this, Eric Chang introduced A new dynamic HMM model for speech recognition, which embeds pitch contour directly as a dynamic hidden state, capturing the continuous frequency modulations inherent in natural speech.
Multi-Scale and Data-Driven Signal Decomposition
If fixed-frequency grids like the standard Fourier transform struggle with non-stationarity, how should we decompose a speech signal? One powerful alternative is to use mathematical frameworks that inherently adapt to multi-scale temporal variations.
Wavelet transforms and scattering networks offer a flexible alternative to the STFT. Joakim Andén in Multiscale Scattering for Audio Classification showed that multiscale scattering representations can capture non-stationary audio dynamics without the information loss typical of fixed-window feature extractors. This is echoed by E. Didiot et al. in A wavelet-based parameterization for speech/music segmentation, who leveraged the compact, multi-resolution nature of wavelets to robustly track transient events and segment highly dynamic audio streams.
Another approach lets the signal decompose itself based on its local oscillations rather than forcing it onto predefined sine or wavelet bases. Empirical Mode Decomposition (EMD) and Variational Mode Decomposition (VMD) break down non-stationary signals into intrinsic mode functions. Prasad Tapkir et al. applied this in Novel Empirical Mode Decomposition Cepstral Features for Replay Spoof Detection, demonstrating that EMD-based features successfully capture the non-linear, non-stationary nuances of genuine versus spoofed speech.
To capture fleeting acoustic cues for emotion recognition, David Hason Rudd et al. in An Extended Variational Mode Decomposition Algorithm Developed Speech Emotion Recognition Performance optimized VMD parameter tuning dynamically, allowing the system to isolate subtle emotional shifts that static filters miss. Furthermore, when frequency modulations dominate, Hui Yin et al. in Adaptive-order fractional Fourier transform features for speech recognition utilized the Fractional Fourier Transform (FrFT), adaptively adjusting the transform order for each frame to accurately track the shifting intonations and frequency-modulated components of non-stationary speech.
Modeling Dynamic Statistical Distributions
Non-stationarity does not just affect frequency and phase; it also causes rapid fluctuations in signal power and statistical distributions. Static statistical models often degrade when applied to speech because a speaker's volume and environmental interactions change second by second.
To address this, Zhaoyi Gu et al. developed an Amplitude Variable Gaussian Mixture Model (AV-GMM) for Independent Vector Analysis in Speech Separation Using Independent Vector Analysis with an Amplitude Variable Gaussian Mixture Model. By introducing a time-varying parameter that tracks the natural power fluctuations of human speech, their model successfully adapts to non-stationary signal estimation without requiring extensive pre-training.
On a broader structural level, Jong Won Shin et al. in Speech probability distribution based on generalized gama distribution proposed using a two-sided generalized gamma distribution. Because it functions as a flexible super-set of standard distributions, it allows for online parameter estimation that adapts dynamically as the speaker's mathematical properties shift over time.
Dynamic Attention and Deep Architectures
Modern deep learning architectures have introduced new ways to handle non-stationarity by replacing static operations with dynamic, data-dependent routing mechanisms.
In tasks like speaker verification, traditional systems rely on global temporal statistics pooling (such as calculating the mean and variance across an entire utterance), which implicitly assumes signal stationarity. Jia Qi Yip et al. challenged this in ACA-Net: Towards Lightweight Speaker Verification using Asymmetric Cross Attention by introducing Asymmetric Cross Attention (ACA). Instead of collapsing the temporal dimension uniformly, ACA uses a fixed-size latent query to dynamically distill variable-length, non-stationary statistical properties, preserving fine-grained acoustic signatures.
For speech enhancement, where different time-frequency regions exhibit highly variable noise levels and structural complexities, uniform processing is highly suboptimal. Xinmeng Xu addressed this in Selector-Enhancer: Learning Dynamic Selection of Local and Non-local Attention Operation for Speech Enhancement. The proposed framework uses a reinforcement learning-based filter to dynamically choose between local and non-local attention operations for different regions of the signal, adapting on the fly to the highly non-stationary nature of both the underlying speech and the surrounding noise.
Key Foundational Work
An Attack on the Problems of Speech Analysis and Synthesis with the Power of an On-Line System (1969) — contributed 1 concept elements to this direction.
An Attack on the Problems of Speech Analysis and Synthesis with the Power of an On-Line System (1969) — contributed 1 concept elements to this direction.