Sparsity-Based Audio Declipping Methods: Selected Overview, New Algorithms, and Large-Scale Evaluation

Clément Gaultier, Srdan Kitic, Rémi Gribonval, Nancy Bertin

TL;DR

This paper introduces a unified framework for audio declipping that encompasses existing and novel sparsity-based methods. Through extensive large-scale evaluations, including listening tests, they provide crucial guidelines for choosing the right declipping approach based on clipping severity and audio content, and call for more perceptually relevant benchmarks.

What Problem Does This Paper Address?

Audio clipping, a form of distortion caused by exceeding signal amplitude limits, severely degrades audio quality and can hinder downstream processing tasks like speech recognition. While significant progress has been made in audio declipping, particularly with sparsity-based techniques, practitioners lack clear guidance on selecting the most effective method for different scenarios. Existing benchmarks often use moderate clipping levels, failing to represent real-world applications with more severe distortions. This paper aims to bridge this gap by proposing a flexible framework, conducting a comprehensive large-scale evaluation, and offering practical guidelines for declipping practice and benchmark design.

Methods

The authors propose a general algorithmic framework for audio declipping that unifies various sparsity-based approaches. This framework leverages two key components:

  1. Sparsity-promoting shrinkage operators: These operators, like hard-thresholding (HT) for plain sparsity and the Persistent Empirical Wiener (PEW) operator for structured (social) sparsity, are used to enforce sparsity in the time-frequency domain.
  2. Generalized projection onto the clipping-consistent data-fidelity constraint: This ensures that the declipped signal adheres to the observed clipped signal in non-clipped regions and respects the clipping threshold in clipped regions.

The framework can accommodate both analysis (cosparse) and synthesis sparse models, as well as plain and structured sparsity priors. They also introduce an adaptive social declipper that dynamically selects the most appropriate time-frequency pattern for different audio segments. The paper provides a unified algorithm (Algorithm 1) that can instantiate various declippers by configuring these components.

Datasets

The paper utilizes several datasets for evaluation:

  • SMALLbox toolbox: Used for initial small-scale experiments and comparisons, particularly for understanding the relationship between clipping measures and perceptual quality.
  • RWC Music Database: A large-scale dataset used for objective experiments, covering "Pop," "Jazz," "Classic" (Vocals, Chamber, Symphonies) genres.
  • TIMIT database: Used for evaluating speech declipping performance.

The authors highlight that existing small-scale benchmarks like SMALLbox often feature moderate clipping levels and call for benchmarks with more perceptually significant clipping levels.

Key Insights and Ideas

  • Perceptual relevance of clipping measures: While clipping threshold and Signal-to-Distortion Ratio (SDR) are common objective measures, their correlation with perceptual quality can be variable, especially for moderate clipping. SDR is generally preferred over clipping threshold as a primary measure.
  • Need for perceptually relevant benchmarks: Existing benchmarks often use moderate clipping levels. The paper advocates for benchmarks with more severe clipping to better reflect real-world scenarios and evaluate the capabilities of modern declipping methods.
  • Unified framework for diverse methods: The proposed framework effectively unifies a wide range of sparsity-based declipping techniques, including plain and structured, analysis and synthesis, and adaptive variants, facilitating systematic comparison.
  • Trade-offs between plain and social sparsity: Plain (co)sparse models tend to perform better in severe clipping conditions, while social sparsity offers advantages in milder conditions and for perceptual quality improvements. However, social sparsity comes with higher computational costs.
  • Adaptive social declippers are perceptually preferred: Despite potential computational overhead, adaptive social declippers were slightly preferred by listeners, even at moderate clipping levels.
  • Computational efficiency is improving: While some methods still have high computational costs, advancements in implementation and parameter tuning suggest that real-time processing is achievable for some plain variants.

Key Results

  • ASDR improvements: The proposed methods, especially plain sparse and adaptive social variants, consistently achieve significant Signal-to-Distortion Ratio (SDR) improvements, often exceeding 8 dB for mild clipping and up to 10 dB for severe clipping.
  • Performance across genres: The paper demonstrates that the proposed methods perform well across various musical genres and speech, with some variations. For instance, plain cosparse methods showed degradation on "Pop" music with saturated guitar sounds, highlighting the importance of dataset characteristics.
  • Perceptual quality gains: Objective perceptual measures (PEAQ, PESQ, STOI) also show significant improvements, with adaptive social methods often leading in perceptual ratings.
  • Listening test validation: Listening tests confirmed that sparsity-based methods significantly outperform older autoregressive interpolation methods, and adaptive social algorithms were perceptually preferred by listeners.
  • Computational performance: Table V provides a detailed comparison of computational performance, showing that some methods are approaching real-time capabilities, especially for less severe clipping.

Limitations and Future Work

  • Limitations:
    • The paper notes that for certain audio content (e.g., intentionally saturated guitar sounds), declipping algorithms might not perform as expected, as their goal is to restore clipped signals, not inherently saturated ones.
  • Detecting social sparsity patterns can be challenging in heavily corrupted signals, potentially impacting the performance of adaptive social methods.
  • Future Work:
    • Further studies are needed to guide the choice of social sparsity patterns, possibly through learning techniques.
  • The authors suggest extending the versatile framework to other audio distortions like denoising and dereverberation, and to multichannel scenarios.
  • The paper calls for the development of more perceptually relevant benchmarks with higher clipping levels.

Summary updated

Abstract

Recent advances in audio declipping have substantially improved the state of the art. Yet, practitioners need guidelines to choose a method, and while existing benchmarks have been instrumental in advancing the field, larger-scale experiments are needed to guide such choices. First, we show that the clipping levels in existing small-scale benchmarks are moderate and call for benchmarks with more perceptually significant clipping levels. We then propose a general algorithmic framework for declipping that covers existing and new combinations of variants of state-of-the-art techniques exploiting time-frequency sparsity: synthesis vs. analysis sparsity, with plain or structured sparsity. Finally, we systematically compare these combinations and a selection of state-of-the-art methods. Using a large-scale numerical benchmark and a smaller scale formal listening test, we provide guidelines for various clipping levels, both for speech and various musical genres. The code is made publicly available for the purpose of reproducible research and benchmarking.
BibTeX
@article{gaultier2021sparsity,
  author = {Clément Gaultier and Srdan Kitic and Rémi Gribonval and Nancy Bertin},
  title = {{Sparsity-Based Audio Declipping Methods: Selected Overview, New Algorithms, and Large-Scale Evaluation}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=EVvmYzlTD8},
}
Figure