When we try to clean a noisy recording—perhaps a voice in a crowded cafe or a faint signal in a radio transmission—we often encounter a frustrating byproduct: "musical noise." Despite the name, this isn't melodic. It is a collection of metallic, tinkling, or chirping artifacts that can be more distracting to human listeners and more damaging to speech recognition systems than the original background hiss.
Understanding why these artifacts appear and how to eliminate them is a central challenge in modern audio signal processing.
The Origin of the Artifacts
The most common way to remove noise is spectral subtraction. In this process, we estimate the "noise profile" of a signal and subtract it from the noisy input. However, noise is rarely constant; it fluctuates randomly.
As Patrick J. Wolfe and Simon J. Godsill explain in Bayesian Estimation of Time-Frequency Coefficients for Audio Signal Enhancement, musical noise arises from random errors in this noise estimation. When the estimated noise power is slightly lower than the actual noise in a specific frequency bin, a small "peak" of residual noise is left behind. Because these errors happen randomly across different frequencies and time frames, these isolated peaks appear and disappear rapidly, creating the sensation of "chirping."
Traditional algorithms often handle "negative energy"—where the estimated noise is higher than the signal—by simply zeroing out those values. Haitian Xu and colleagues note in Spectral subtraction with full-wave rectification and likelihood controlled instantaneous noise estimation for robust speech recognition that this "zero-masking" creates harsh discontinuities in the sound spectrum, further contributing to the artificial, watery quality of the output.
The Masking Trap: Binary vs. Soft Decisions
A major source of musical noise in modern source separation is the use of binary masks. A binary mask makes a hard, "all-or-nothing" decision for every point in a spectrogram: either this pixel belongs to the voice, or it belongs to the noise.
Cosimo Riday and Richard H. R. Hahnloser argue in Monaural Source Separation Using a Random Forest Classifier that these rigid assignments cause spectral artifacts at the boundaries of sound clusters. To solve this, researchers like Emad M. Grais have moved toward soft masking. In Combining Mask Estimates for Single Channel Audio Source Separation Using Deep Neural Networks, they demonstrate that using a "combiner" network to merge binary and soft masks can retain the high interference rejection of binary masks while smoothing out the unnatural distortions that lead to musical noise.
Smoothing Through Time and Frequency
If musical noise is caused by rapid, random fluctuations, the intuitive solution is to smooth those fluctuations out. This can be done in several ways:
- Temporal Smoothing: By ensuring that the "gain" (the amount of suppression) doesn't change too abruptly between consecutive frames, we can prevent the rapid flickering of spectral peaks. Dong Yu highlights this in Improvements on Mel-Frequency Cepstrum Minimum-Mean-Square-Error Noise Suppressor for Robust Speech Recognition, showing that temporal gain smoothing is essential for maintaining the integrity of features used in automated recognition.
- Adaptive Segmentation: Standard audio processing uses fixed-length windows (e.g., 20ms). Sriram Srinivasan proposes in Speech enhancement using adaptive time-domain segmentation that we should instead segment audio into its longest "stationary" chunks. This prevents noise peaks from jumping around between arbitrary frame boundaries, effectively silencing the "chirp."
- Statistical Priors: Instead of guessing the noise, we can use a model of what clean speech should look like. Samy Elshamy and Tim Fingscheidt describe an iterative estimator in An iterative speech model-based a priori SNR estimator that matches noisy signals against a Gaussian Mixture Model (GMM) of clean speech, ensuring the output stays within the realm of natural-sounding audio.
Mimicking the Human Ear
Perhaps the most effective way to eliminate musical noise is to stop treating audio as a mathematical matrix and start treating it as something heard by a human.
The human ear doesn't perceive all frequencies equally. Jürgen Tchorz and Birger Kollmeier pioneered neurophysiologically-motivated SNR estimation in Noise Suppression Based on Neurophysiologically-motivated SNR Estimation for Robust Speech Recognition, using amplitude modulation spectrograms to mimic mammalian auditory processing.
Similarly, the AUDlet filter bank, introduced by Thibaud Necciari and Peter Balazs in A Perceptually Motivated Filter Bank with Perfect Reconstruction for Audio Signal Processing, provides a mathematical framework that mimics the non-linear frequency resolution of the ear. By aligning the processing "bins" with the ear's critical bands, these methods ensure that any residual noise is "masked" by the speech itself, making it perceptually invisible even if it is mathematically present.
Modern Neural Refinement
The latest frontier involves training deep neural networks (DNNs) to optimize for human-centric quality scores directly. Yuma Koizumi and colleagues in DNN-Based Source Enhancement to Increase Objective Sound Quality Assessment Score use reinforcement learning to reward networks for producing high PESQ (Perceptual Evaluation of Speech Quality) scores. This forces the model to learn that musical noise is a "high-cost" error, leading to cleaner, more natural results than traditional distance-based training.
Finally, models like TokenSplit, developed by Hakan Erdoğan and John R. Hershey, move away from spectrogram masking entirely. In TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition, they treat audio as a sequence of discrete tokens. By reconstructing speech from these high-level "semantic" units, the system bypasses the low-level spectral fluctuations that cause musical noise altogether.