Audio Signal Enhancement with Learning from Positive and Unlabelled Data

Nobutaka Ito, Masashi Sugiyama

Introduction

Researchers from the University of Tokyo have developed PULSE, a novel framework that performs audio signal enhancement (SE) without requiring pairs of "clean" and "noisy" recordings. By leveraging Positive and Unlabelled (PU) learning, the model learns to suppress noise using only non-parallel data—recordings of pure noise and recordings of noisy signals—bringing deep learning-based enhancement closer to real-world feasibility.

The Challenge of Parallel Data

Most modern speech enhancement systems rely on supervised learning. To train these models, researchers typically require "parallel data": a recording of clean speech and an identical recording of that same speech drowned in noise.

Comparison between conventional supervised learning and the proposed PU learning framework.

Because it is physically impossible to record both simultaneously in the same space due to microphone crosstalk, researchers usually synthesize this data by digitally adding noise to clean clips. This creates two major hurdles:

  1. Domain Mismatch: Synthetic data rarely captures the complex acoustics of real-world environments perfectly, leading to poor performance when the model meets "real" noise.
  2. Inaccessibility: For many sound sources—like specific musical instruments or rare bird calls—clean, noise-free "ground truth" recordings simply do not exist.

The authors address these gaps by asking: Can we train a high-quality enhancement model using only the data we can easily record in the wild?

Methods: The PULSE Framework

The core innovation is the application of PU (Positive and Unlabelled) Learning to the audio domain. In standard binary classification, you have Positive (P) and Negative (N) samples. In PU learning, you only have Positive samples and "Unlabelled" (U) samples (which contain a mix of P and N).

The PULSE workflow: extracting noise patches (P) and unlabeled patches (U) for PU learning classification.
  • Positive (P): The authors define this as signal inactivity (pure noise). These are easy to record when a person stops talking.
  • Unlabelled (U): This is the noisy signal (speech + noise).

The PULSE system treats spectrogram patches as data points. A Convolutional Neural Network (CNN) is trained to classify whether a specific time-frequency patch is "Noise" (P) or "Signal Activity" (N). Once trained, the CNN acts as a mask provider: it identifies noise-dominated components and suppresses them, leaving the enhanced signal behind.

Technical Refinements

Two key technical refinements make this work for audio:

  1. Weighted Sigmoid Loss: Standard PU learning treats all samples equally. However, in audio, loud components are more important for perceived quality than quiet ones. The authors introduced a loss function weighted by the magnitude of the spectrogram, ensuring the model prioritizes accuracy where the sound is most prominent.
  2. Non-negative Empirical Risk (nnPU): PU learning math can sometimes result in "negative risk" during training, which destabilizes deep networks. The authors used a non-negative risk estimator to keep training stable.

Datasets and Evaluation

The study used synthetic mixtures to allow for precise quantitative evaluation (which requires a "clean" reference), though the training process itself only used the data in a non-parallel fashion. The researchers used the TIMIT acoustic-phonetic corpus for speech and the DEMAND dataset for diverse real-world noise environments (kitchens, offices, traffic, etc.).

Key Results

The researchers evaluated performance using SI-SNRi (Scale-Invariant Signal-to-Noise Ratio improvement), measured in decibels (dB).

Method SI-SNRi (dB) Data Requirement
PULSE (Proposed) 14.62 Non-parallel (Noisy + Noise)
Supervised Learning 15.86 Parallel (Noisy + Clean)
MixIT (Prior SOTA) 12.19 Non-parallel
PULSE w/o Weighted Loss 9.30 Non-parallel

PULSE significantly outperformed MixIT (a popular unsupervised method) and came remarkably close to the performance of Supervised Learning, despite never seeing a single "clean" speech clip during training.

Conclusion and Future Work

The results show that you don't actually need the "clean" signal to learn what a clean signal looks like; the statistical difference between "pure noise" and "noise + signal" is sufficient for a CNN to learn enhancement. Future work will focus on testing PULSE on real-world recordings using non-intrusive quality metrics and applying the PU learning framework to more sophisticated architectures like Transformers or U-Nets.

Summary updated

BibTeX
@inproceedings{ito2023audio,
  author = {Nobutaka Ito and Masashi Sugiyama},
  title = {{Audio Signal Enhancement with Learning from Positive and Unlabelled Data}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095988},
}
Figure