It Is Likely That Your Loss Should be a Likelihood

Mark Hamilton, Evan Shelhamer, William T. Freeman

TL;DR

Standard loss functions like mean-squared-error and cross-entropy implicitly fix probability distribution parameters (such as variance or temperature) to constants. This paper demonstrates that treating these "likelihood parameters" as learnable variables optimized jointly with the model enables automatic loss adaptation, robust outlier handling, and superior model calibration without manual hyperparameter tuning.

What Problem Does This Paper Address?

Choosing the right loss function and regularization strength is critical to machine learning performance, but it typically requires tedious manual hyperparameter tuning, such as grid searching for weight decay coefficients or loss-balancing weights.

Standard losses like mean-squared-error (MSE) or cross-entropy assume fixed underlying probability distributions (e.g., a Normal distribution with a fixed variance of $\sigma=1$). This rigidity makes models highly sensitive to outliers, prone to miscalibration (overconfidence), and inflexible when dealing with data-dependent uncertainty (heteroskedasticity). The authors address this gap by framing loss optimization as full maximum likelihood estimation, where the shape and scale of the loss function itself are learned dynamically alongside the model parameters.

Methods

The authors propose Full Likelihood Optimization, which jointly learns model parameters ($\theta$) and likelihood parameters ($\phi$). They systematically explore three types of likelihoods: Normal (variance $\sigma$), Softmax (temperature $\tau$), and the generalized robust loss $\rho$ from Barron (2019).

They categorize how these likelihood parameters can be conditioned:

  1. Global: Task-dependent but constant across all data points.
  2. Data: Independent parameters assigned to each specific training data index (useful for training curricula but unavailable at test time).
  3. Predicted: Outputted by a sub-network based on the input data, allowing test-time adaptation.
Figure 1: Optimizing likelihood parameters adapts the loss without manual hyperparameter tuning to balance accuracy and certainty.

Additionally, they introduce Adaptive Regularization (Dynamic Lasso/Ridge and Multi-Lasso) by treating regularization weights as learnable scale parameters of normal or Laplace priors, eliminating the need to cross-validate the regularization coefficient $\lambda$. To fix optimization instabilities (e.g., gradients exploding near $\tau=0$), they introduce a novel shifted softplus function to smoothly bound the learned parameters.

Datasets

The paper evaluates these methods across several distinct domains:

  • Image Classification & Generation: CIFAR-10, CIFAR-100, SVHN, ImageNet, and CelebA (for Variational Autoencoders).
  • Tabular Regression: Eight standard UCI datasets (including Crime, Kinematics, Bank, Wine, MPG, CPU, Soil, and Fried).
  • Outlier Detection: The Outlier Detection Data Sets (ODDS) benchmark, containing 22 distinct datasets with ground-truth outlier labels.

Key Insights and Ideas

  • The Uncertainty vs. Credit Trade-off: Learning the scale parameter isn't "cheating." A model can lower its penalty for a large error by increasing its uncertainty (e.g., higher variance $\sigma$), but it pays a price via a normalization penalty ($\log \sigma$). This forces the model to balance accuracy and certainty automatically.
  • Outlier Detection for Free: When a model predicts its own likelihood parameters (like temperature or variance), these parameters naturally act as outlier scores. High predicted variance or temperature perfectly flags noisy, blurry, or mislabeled data points.
  • Gradient-Based Hyperparameter Tuning: Instead of running expensive grid searches for $L_1$ or $L_2$ regularization strengths, backpropagation can directly optimize the prior's scale parameters, finding the ideal regularization strength dynamically during training.

Key Results

  • Robust Modeling: On a VAE reconstruction task using CelebA, predicting pixel-wise robust loss parameters achieved the lowest MSE (222.3) while maintaining a tiny memory footprint (<1MB) compared to data-conditioned approaches, which required 4.4GB of storage.
  • Outlier Detection: Simple autoencoders augmented with learned scale parameters (PCA+S and AE+S) achieved state-of-the-art median AUCs of 0.868 and 0.846 on the 22 ODDS benchmark datasets, outperforming established methods like Isolation Forests (0.823) and One-Class SVMs (0.814).
  • Re-calibration: The proposed Linear Scaling (LS) and Linear Feature Scaling (LFS) methods significantly reduced Expected Calibration Error (ECE) across deep architectures (ResNet-50, DenseNet-121), outperforming standard Global Temperature Scaling.

Limitations and Future Work

  • Ambiguity in Uncertainty Attribution: For classifiers with learned temperatures, the model can choose to store uncertainty either by reducing logit magnitudes or by increasing the temperature parameter. This split can make interpreting temperature as a pure outlier detector difficult if the parameters train at different rates.
  • Architecture Agnosticism: While the approach is model-agnostic, the authors note that future work could explore tighter integration with domain-specific architectures tailored specifically for text, audio, or complex structured data.

Summary updated

Abstract

Many common loss functions such as mean-squared-error, cross-entropy, and reconstruction loss are unnecessarily rigid. Under a probabilistic interpretation, these common losses correspond to distributions with fixed shapes and scales. We instead argue for optimizing full likelihoods that include parameters like the normal variance and softmax temperature. Joint optimization of these "likelihood parameters" with model parameters can adaptively tune the scales and shapes of losses in addition to the strength of regularization. We explore and systematically evaluate how to parameterize and apply likelihood parameters for robust modeling, outlier-detection, and re-calibration. Additionally, we propose adaptively tuning $L_2$ and $L_1$ weights by fitting the scale parameters of normal and Laplace priors and introduce more flexible element-wise regularizers.
BibTeX
@misc{hamilton2020it,
  author = {Mark Hamilton and Evan Shelhamer and William T. Freeman},
  title = {{It Is Likely That Your Loss Should be a Likelihood}},
  year = {2020},
  eprint = {2007.06059},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.06059},
}
Figure