Eliminating Musical Noise Artifacts in Speech Enhancement

155 concepts132 papers

When we attempt to clean up a noisy audio recording, we often inadvertently introduce a new, equally distracting problem. Instead of a pristine voice, the processed audio sounds as if it is trapped underwater, punctuated by artificial, metallic, or watery chirping sounds. This phenomenon is known as musical noise, and it is one of the most persistent artifacts in digital signal processing and speech enhancement.

To understand why musical noise happens—and how modern algorithms eliminate it—we have to look at how sound is manipulated in the time-frequency domain.

The Origin of the Chirp

Most audio enhancement algorithms operate by breaking a continuous sound wave into short, overlapping frames and applying the Short-Time Fourier Transform (STFT). This converts the audio into a spectrogram: a two-dimensional grid where one axis represents time, the other represents frequency, and each cell (or bin) contains the energy level of that specific frequency at that specific moment.

Traditional enhancement techniques, such as spectral subtraction or ideal binary masking, attempt to separate speech from background noise by applying a mathematical threshold to this grid. If the energy in a specific bin falls below the estimated noise floor, the algorithm assumes it is noise and attenuates or zeroes it out entirely.

As noted by Li Xu et al. and Wenye Ma et al., this hard-thresholding strategy creates a highly discontinuous, "jagged" spectral landscape. Because background noise fluctuates randomly, some frequency bins will randomly spike just above the threshold, while their immediate neighbors are completely zeroed out.

This leaves behind isolated "islands" of spectral energy scattered randomly across the time-frequency grid:

Energy Level
   ^
   |       _/\_                 _/\_          <-- Isolated Spectral Islands
   |______/____\_______________/____\________
   +------------------------------------------> Frequency

When this modified spectrogram is converted back into a continuous time-domain waveform via the Inverse DFT, these isolated spectral islands manifest as short-lived, random sinusoids. To the human ear, a rapid succession of random, isolated pure tones does not sound like background hiss; it sounds like a faint, metallic ringing—musical noise.

Statistical and Structural Mismatches

Why do these thresholds fail so predictably? The root cause is often a mismatch between mathematical assumptions and the true nature of sound.

Inaccurate Statistical Modeling

Classical speech enhancement algorithms frequently assume that speech and noise follow a Gaussian (normal) distribution in the frequency domain. However, as Joon-Hyuk Chang demonstrates in Complex laplacian probability density function for noisy speech enhancement, speech is highly non-Gaussian and is much more accurately modeled by a complex Laplacian distribution. When an algorithm relies on an incorrect Gaussian assumption, its noise suppression becomes inconsistent across frequency bins, directly generating the fluctuating spectral islands that cause musical noise.

The Rigidity of Binary Decisions

Binary masking techniques force a hard, binary choice: a time-frequency bin contains either 100% target signal or 100% noise. Emad M. Grais et al. point out that while binary masks offer excellent interference rejection, this mathematical rigidity inherently produces severe spectral artifacts because it completely ignores the natural overlap between speech and environmental noise.

Strategies for Mitigation

To eliminate musical noise, researchers have shifted away from hard thresholds toward methods that enforce smoothness, leverage human perception, or optimize for structural continuity.

1. Perceptual Masking and Comfort Noise

Instead of trying to eliminate 100% of the background noise—which inevitably creates isolated spectral islands—perceptual approaches attempt to "hide" the residual noise behind the human ear's natural limitations.

Gwo-hwa Ju et al. and Ning Cheng et al. introduced psychoacoustic models directly into the filtering process. By calculating the human auditory masking threshold, these algorithms ensure that any leftover noise is shaped to fall below what the listener can consciously perceive. Rather than a jarring silence punctuated by metallic chirps, the listener hears a smooth, natural "comfort noise" profile that the brain naturally filters out.

2. Mathematical Smoothness via Convex Optimization

Another approach tackles the problem by enforcing continuity directly within the optimization framework. To prevent the formation of isolated spectral holes, Wenye Ma et al. proposed shifting from the frequency domain to a time-domain convex optimization framework. By applying $L_1$ regularization and the fast Split Bregman method, they estimate sparse filters that smooth out the sharp discontinuities, preventing the creation of the jagged spectral edges that cause musical noise.

3. Component-Aware Neural Objectives

In modern deep learning architectures, networks trained on global objectives like Mean Squared Error (MSE) often overfit by aggressively cutting out noise at the expense of signal naturalness, resulting in robotic or musical artifacts.

To combat this, Ziyi Xu et al. developed a "Components Loss" function. By explicitly separating the training objective into distinct terms for speech preservation, noise suppression, and residual noise naturalness, the network learns to suppress background noise smoothly. This ensures that the residual signal maintains a perceptually pleasant texture rather than degenerating into artificial distortions.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_musical_noise_artifact_a079a062,
  title = {musical noise artifact},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/musical-noise-artifact/rdr_a079a0629ed04675b4ff87d52a65fde8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_musical_noise_artifact_a079a062_v1,
  title = {musical noise artifact},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/musical-noise-artifact/rdr_a079a0629ed04675b4ff87d52a65fde8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake