Robustness enhancement in neural networks with alpha-stable training noise

Ercan Engin Kuruoglu, PhD Cantab, Prof, Ieee Fellow, Xueqiong Yuan, Jipeng Li, Ercan Engin KuruogluDigit. Signal Process 2025 · DOI

TL;DR

This paper challenges the conventional use of Gaussian noise for training robust neural networks. By introducing alpha-stable noise, a type of impulsive noise found in nature, the authors demonstrate significantly improved robustness, especially against impulsive corruptions. Their proposed method of replacing Gaussian noise with alpha-stable noise during data augmentation offers a more effective and generalizable approach to building resilient deep learning models.

What Problem Does This Paper Address?

Deep learning models are increasingly deployed in real-world scenarios where data is often imperfect, corrupted by noise, or subject to various perturbations. This lack of robustness can lead to unreliable performance. A common technique to enhance robustness is data augmentation, specifically by adding Gaussian noise to training data. However, the authors argue that Gaussian noise might not be sufficient to capture the full spectrum of real-world noise, particularly impulsive noise, which is characterized by abrupt peaks or occasional bursts. Such impulsive noise is prevalent in various natural phenomena and sensor data (e.g., images from noisy sensors, ECG signals, radar data). The paper aims to investigate whether using a more realistic noise model, like alpha-stable noise, can lead to better robustness than the standard Gaussian noise approach.

Methods

The core of this paper is the proposal and evaluation of a novel data augmentation method. Instead of adding Gaussian noise to training data, the authors introduce alpha-stable noise. Alpha-stable distributions are a generalization of the Gaussian distribution, capable of modeling heavy-tailed, impulsive noise. The key parameter, $\alpha$ (the characteristic exponent), controls the impulsiveness: smaller $\alpha$ values lead to heavier tails and more impulsive noise.

The authors explore several strategies:

  • Single a-stable noise training: Training models with a single type of alpha-stable noise.
  • Combined noise training:
    • Multiple noise: Training with a mix of different alpha-stable noise types.
  • Mixture noise: Training with noise sampled from an alpha-stable mixture model.

They compare the performance of models trained with these alpha-stable noise strategies against models trained with clean data and Gaussian noise, across various datasets and network architectures.

Datasets

The study utilizes a range of datasets to demonstrate the generality of their approach:

  • Image Datasets:
    • MNIST (handwritten digits)
    • CIFAR10 (color images)
  • Time Series Datasets:
    • ECG200 (electrocardiograms)
    • LIBRAS (Brazilian sign language)
  • Benchmark Corrupted Datasets:
    • MNIST-C
    • CIFAR10-C These benchmark datasets are corrupted versions of MNIST and CIFAR10, encompassing a wide array of perturbations beyond just Gaussian and impulse noise, such as blur, spatter, and weather conditions.

Key Insights and Ideas

  • Gaussian Noise is Not Always Enough: The paper strongly suggests that relying solely on Gaussian noise for data augmentation might be a suboptimal choice when dealing with real-world data that often exhibits impulsive characteristics.
  • Alpha-Stable Noise Mimics Real-World Impulsiveness: Alpha-stable distributions provide a more accurate mathematical model for impulsive noise, which is frequently encountered in various natural and technical domains.
  • "Multiple Noise" Training is Highly Effective: Training with a combination of different alpha-stable noise types (the "multiple noise" scenario) consistently yields superior robustness compared to single alpha-stable noise or Gaussian noise. This suggests that exposing the model to a diverse set of impulsive noise patterns during training is key.
  • Robustness Across Diverse Corruptions: The proposed alpha-stable noise augmentation method not only improves resilience to impulsive noise but also demonstrates effectiveness against other types of corruptions, indicating its broad applicability.
  • Sparser Models: Models trained with alpha-stable noise tend to exhibit sparser parameter distributions, which the authors link to enhanced generalization capabilities.

Key Results

  • Models trained with alpha-stable noise, particularly the "multiple noise" strategy, consistently outperform models trained with Gaussian noise in terms of accuracy, especially when tested on data corrupted by impulsive noise.
  • For instance, on the MNIST-C dataset, the "multiple noise" strategy achieved an average accuracy of 51.46%, significantly higher than Gaussian noise (45.59%) and clean data (42.61%). Similar improvements were observed across other datasets and models.
  • Even when tested on clean data or data with Gaussian noise, models trained with alpha-stable noise generally maintained comparable or better performance than those trained with Gaussian noise, demonstrating its generalizability.
  • The paper recommends using Cauchy noise ($\alpha=1$) or multiple alpha-stable noise as effective data augmentation strategies.

Limitations and Future Work

  • Clipping Artifacts: For image data, when using very small $\alpha$ values (below 0.5), pixel values can become clipped, potentially affecting the reliability of results. The study limits $\alpha$ to 0.5 and above for image data.
  • Optimal $\gamma$ Determination: While the study explores various $\alpha$ values, determining the optimal dispersion parameter ($\gamma$) for different datasets and noise levels remains a challenge. The authors suggest that adaptive approaches to noise perturbation levels, as seen in recent work, could be a promising direction.
  • Limited to Classification: The current research focuses on classification tasks. Future work could explore the impact of alpha-stable noise on other supervised learning tasks like regression and object detection.
  • Broader Data Types: The study primarily uses image and time series data. Extending the investigation to other data modalities like audio, video, and text is suggested for future research.

Summary updated

BibTeX
@article{fellow2025robustness,
  author = {Ercan Engin Kuruoglu, PhD Cantab, Prof, IEEE Fellow and Xueqiong Yuan and Jipeng Li and Ercan Engin Kuruoglu},
  title = {{Robustness enhancement in neural networks with alpha-stable training noise}},
  journal = {Digit. Signal Process},
  year = {2025},
  doi = {10.1016/j.dsp.2024.104778},
}

Loading related content...

Figure