Taming the Long Tail of Deep Probabilistic Forecasting

Jedrzej Kozerawski, Mayank Sharan, Rose Yu

Introduction

In safety-critical domains such as autonomous driving and energy management, "average" performance is often a deceptive metric. While a trajectory predictor might achieve low mean error, failing to anticipate a rare, sharp maneuver can lead to catastrophic outcomes. This is known as the long-tail problem: a small subset of samples is disproportionately difficult to forecast, creating a "fat tail" in the error distribution.

Standard deep learning models typically optimize for Mean Squared Error (MSE), which prioritizes the frequent "easy" majority. Research suggests that this long-tail behavior is not merely a data artifact but is often model-induced, where neural networks overfit simple patterns and neglect complex outliers.

Proposed Methodology: Tail-Aware Loss Functions

To address this, we propose modifying the training objective to explicitly account for the "tailedness" of the error distribution through two novel approaches:

1. Kurtosis Loss

Kurtosis measures the "peakedness" or "tailedness" of a distribution. By incorporating the fourth statistical moment into the loss function, the model is heavily penalized for producing extreme outliers, forcing the network to attend to the tail of the distribution.

2. Pareto Loss

Drawing from Extreme Value Theory (EVT), we fit a Generalized Pareto Distribution (GPD) to the model's errors. Two variants are introduced:

  • Pareto Loss Margin (PLM): An additive regularization term that serves as a penalty margin for difficult examples.
  • Pareto Loss Weighted (PLW): A reweighting scheme that scales individual sample losses based on their position in the estimated error distribution.

Unlike Focal Loss, these methods are tailored for the continuous, unbounded nature of forecasting errors.

Experimental Setup

The effectiveness of these loss functions was evaluated across diverse datasets, ranging from real-world time series to spatiotemporal trajectories:

  • Real-World Time Series: Electricity consumption and traffic occupancy (UCI repository).
  • Spatiotemporal Trajectories: Pedestrian movement (ETH-UCY) and autonomous driving (nuScenes).
  • Synthetic Benchmarks: Sine, Gaussian, and Pareto 1D series to isolate model-induced tail effects.
Figure 4: Ground truth value distributions across four different datasets, illustrating the inherent long-tail nature of the data used for evaluation.

Key Insights and Results

Our experiments yielded several critical findings:

Figure 1: Comparison of error distributions and trajectory predictions. The main plot highlights the "long tail" of errors, while the inset demonstrates how the Pareto Loss Margin (ours) more accurately captures ground truth trajectories compared to baselines.
  • Model-Induced Tails: Even on smooth synthetic data (e.g., Sine waves), deep models like DeepAR produce long-tailed errors, confirming that specific loss functions are necessary to "tame" the learning process.
  • Regularization Stability: For extreme tails, additive penalties (PLM) proved more stable than multiplicative reweighting (PLW), which can lead to erratic training gradients.
  • Error Accumulation: Long-tail issues become more severe as the prediction horizon increases, as errors compound over time.
  • Performance Gains: On the Electricity dataset, Kurtosis loss reduced Maximum Error from 4.15 to 3.27. In the nuScenes trajectory task, Pareto Loss (PLM) significantly improved the Final Displacement Error (FDE) for the most difficult 1% of cases (VaR 99) without degrading average performance.

Conclusion

By shifting the focus from average error to the statistical properties of the error distribution, Kurtosis and Pareto losses provide a robust framework for improving reliability in probabilistic forecasting. Future work will explore automated hyperparameter tuning and applications in high-dimensional tasks like video frame prediction.

Summary updated

BibTeX
@inproceedings{kozerawski2023taming,
  author = {Jedrzej Kozerawski and Mayank Sharan and Rose Yu},
  title = {{Taming the Long Tail of Deep Probabilistic Forecasting}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=fvvcpsEl3Z6},
}
Figure