On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression

Jun Qi, Jun Du, Sabato Marco Siniscalchi, Xiaoli Ma, Chin-Hui Lee

Abstract

In this paper, we exploit the properties of mean absolute error (MAE) as a loss function for the deep neural network (DNN) based vector-to-vector regression. The goal of this work is two-fold: (i) presenting performance bounds of MAE, and (ii) demonstrating new properties of MAE that make it more appropriate than mean squared error (MSE) as a loss function for DNN based vector-to-vector regression. First, we show that a generalized upper-bound for DNN-based vector-to-vector regression can be ensured by leveraging the known Lipschitz continuity property of MAE. Next, we derive a new generalized upper bound in the presence of additive noise. Finally, in contrast to conventional MSE commonly adopted to approximate Gaussian errors for regression, we show that MAE can be interpreted as an error modeled by Laplacian distribution. Speech enhancement experiments are conducted to corroborate our proposed theorems and validate the performance advantages of MAE over MSE for DNN based regression.

Introduction

In deep neural network (DNN) regression, Mean Squared Error (MSE) has long been the default loss function. This preference stems from its historical association with Gaussian error modeling and its favorable optimization properties, such as smooth gradients. However, empirical observations in fields like speech enhancement often show that Mean Absolute Error (MAE) yields superior performance, particularly in noisy environments.

This paper addresses the lack of a rigorous theoretical framework explaining why MAE generalizes better than MSE in vector-to-vector regression. By examining performance bounds and robustness through learning theory, we provide a theoretical defense of MAE, highlighting its Lipschitz continuity and its alignment with the Laplacian distribution.

Theoretical Framework

The superiority of MAE is established through three primary theoretical pillars:

1. Generalization Bounds and Lipschitz Continuity

A critical finding is that MAE is 1-Lipschitz continuous. Using Rademacher complexity to measure the richness of the hypothesis space, we prove that this property allows for a stable upper bound on estimation error. In contrast, MSE lacks Lipschitz continuity, making its generalization error theoretically harder to bound and potentially more volatile.

2. Robustness to Additive Noise

We derive a theorem demonstrating that for MAE-based objectives, the impact of additive noise on the output is bounded by the magnitude of the noise itself. This mathematical guarantee of stability is absent for MSE, which, due to its squaring operation, is disproportionately sensitive to outliers and noise spikes.

3. Probabilistic Interpretation

While MSE is linked to the Gaussian distribution, we establish a formal connection between MAE and the Laplacian distribution. In many real-world regression tasks, such as speech processing, errors often exhibit heavy-tailed characteristics that are more accurately modeled by a Laplacian distribution than a Gaussian one.

Experimental Setup

To validate these theoretical claims, we conducted speech enhancement experiments using a deep feed-forward network with ReLU activation.

Dataset

The Edinburgh noisy speech database was utilized:

  • Training: 23,075 clean utterances synthesized with 40 noise types at four SNR levels (15dB, 10dB, 5dB, 0dB).
  • Testing: 824 noisy utterances across 20 unseen conditions, with SNR values ranging from 17.5dB to 2.5dB.

Results and Discussion

The empirical results consistently favor the MAE-optimized model (DNN-MAE) over the MSE-optimized model (DNN-MSE) across all evaluated metrics.

Error Metrics

As shown in the table below, DNN-MAE achieved a lower MAE (0.7812) than DNN-MSE (0.8278). Notably, the DNN-MAE model also achieved a lower MSE (0.7954) than the model specifically optimized for MSE (0.8371), suggesting that MAE provides a more robust optimization path even for MSE-based evaluation.

Comparison of MAE and MSE metrics for models trained with different loss functions.

Speech Quality and Intelligibility

The performance advantages extend to perceptual metrics. DNN-MAE outperformed DNN-MSE in both speech quality (PESQ) and intelligibility (STOI).

Speech quality (PESQ) and intelligibility (STOI) comparison between DNN-MAE and DNN-MSE.

These results confirm that the theoretical stability and noise robustness of MAE translate into tangible improvements in signal reconstruction and noise suppression.

Conclusion

This work provides both theoretical and empirical evidence for the advantages of MAE over MSE in deep vector-to-vector regression. By leveraging Lipschitz continuity, we ensure tighter generalization bounds and greater robustness to noise. While MSE remains a common choice, our findings suggest that MAE, with its natural alignment to Laplacian error distributions, is a more appropriate loss function for complex regression tasks like speech enhancement.

Summary updated

Abstract

In this paper, we exploit the properties of mean absolute error (MAE) as a loss function for the deep neural network (DNN) based vector-to-vector regression. The goal of this work is two-fold: (i) presenting performance bounds of MAE, and (ii) demonstrating new properties of MAE that make it more appropriate than mean squared error (MSE) as a loss function for DNN based vector-to-vector regression. First, we show that a generalized upper-bound for DNN-based vector- to-vector regression can be ensured by leveraging the known Lipschitz continuity property of MAE. Next, we derive a new generalized upper bound in the presence of additive noise. Finally, in contrast to conventional MSE commonly adopted to approximate Gaussian errors for regression, we show that MAE can be interpreted as an error modeled by Laplacian distribution. Speech enhancement experiments are conducted to corroborate our proposed theorems and validate the performance advantages of MAE over MSE for DNN based regression.
BibTeX
@article{qi2020mean,
  author = {Jun Qi and Jun Du and Sabato Marco Siniscalchi and Xiaoli Ma and Chin-Hui Lee},
  title = {{On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression}},
  journal = {IEEE Signal Processing Letters},
  year = {2020},
  url = {https://openreview.net/forum?id=8sIj0YOaq9},
}
Figure