TL;DR
This paper provides a theoretical and empirical defense of Mean Absolute Error (MAE) over the standard Mean Squared Error (MSE) for deep learning regression. By proving that MAE possesses the property of Lipschitz continuity—which MSE lacks—the authors demonstrate that MAE offers tighter generalization bounds, better robustness to noise, and a natural mathematical alignment with the Laplacian distribution.
What Problem Does This Paper Address?
In the world of deep neural network (DNN) regression, Mean Squared Error (MSE) is the "default" loss function, largely due to its historical roots in Gaussian error modeling and its smooth gradient properties. However, researchers have long observed that MAE often yields better results in practice, especially in noisy environments like speech enhancement.
The authors identify a significant gap: while we have empirical evidence that MAE works well, we lack a rigorous theoretical framework explaining why it generalizes better than MSE in the context of deep vector-to-vector regression. They set out to bridge this gap by examining performance bounds and robustness through the lens of learning theory.
Methods
The paper approaches the problem through three main theoretical pillars and one empirical validation:
- Generalization Bounds: The authors utilize Rademacher complexity to measure the richness of the hypothesis space. They prove that because MAE is 1-Lipschitz continuous, it allows for a stable upper bound on the estimation error. They contrast this with MSE, which is not Lipschitz continuous, making its generalization error harder to bound theoretically.
- Robustness Analysis: They derive a theorem showing that for MAE-based objectives, the impact of additive noise on the output is bounded by the magnitude of the noise itself. This mathematical guarantee of "stability" is absent for MSE.
- Probabilistic Interpretation: The authors establish a formal connection between MAE and the Laplacian distribution, comparing it to the well-known link between MSE and the Gaussian distribution. They argue that the Laplacian model is often a more accurate representation of real-world regression errors.
- Empirical Validation: They train a deep feed-forward network (ReLU-based) for speech enhancement, comparing models optimized with MAE versus MSE.
Datasets
The authors used the Edinburgh noisy speech database.
- Training: 23,075 clean utterances synthesized with 40 different noise types (e.g., kitchen, office, subway, street) at four SNR levels (15dB, 10dB, 5dB, 0dB).
- Testing: 824 noisy utterances across 20 conditions not seen during training, including SNR values ranging from 17.5dB down to 2.5dB.
Key Insights and Ideas
- The Lipschitz Advantage: The "aha moment" is the proof that MAE is 1-Lipschitz continuous. In simple terms, this means the function doesn't "explode" or change too abruptly. This property is the key requirement for applying many fundamental theorems in learning theory that guarantee a model will perform similarly on unseen data as it did on training data.
- MSE's Sensitivity: Because MSE squares the error, it is disproportionately sensitive to outliers or large noise spikes. The paper mathematically demonstrates that this lack of Lipschitz continuity means MSE cannot guarantee the same level of robustness against additive noise that MAE can.
- Laplacian vs. Gaussian: By linking MAE to the Laplacian distribution, the authors highlight that MAE is essentially a "Maximum Likelihood" estimator for errors that follow a heavier-tailed distribution. In many signal processing tasks like speech, errors are rarely perfectly Gaussian, making the Laplacian (and thus MAE) a more natural fit.
Key Results
The experimental results on the speech enhancement task consistently favored MAE across all metrics:
- Error Metrics: The model trained on MAE (DNN-MAE) achieved a lower MAE score (0.7812) compared to the MSE-trained model (0.8278). Interestingly, the DNN-MAE model even achieved a lower MSE score than the model actually optimized for MSE (0.7954 vs 0.8371).
- Speech Quality (PESQ): DNN-MAE scored 2.93, outperforming DNN-MSE’s 2.85.
- Intelligibility (STOI): DNN-MAE scored 0.8509, superior to DNN-MSE’s 0.8317.
These results confirm that the theoretical bounds derived in the paper translate into tangible improvements in signal reconstruction and noise suppression.
Limitations and Future Work
While the paper provides a strong case for MAE, there are a few areas for further exploration:
- Optimization Challenges: MAE is non-differentiable at zero. While modern optimizers handle this via sub-gradients, the paper doesn't deeply explore if this causes convergence issues in much deeper or more complex architectures (like Transformers).
- Activation Functions: The proofs rely on the Lipschitz continuity of the network itself, which holds for ReLU. The authors note this, but future work could extend these proofs to other modern activation functions.
- Task Specificity: The empirical evidence is focused on speech enhancement. While the theory is general, testing these bounds on other vector-to-vector tasks (like image-to-image translation) would further strengthen the claims.
Summary updated