Uncertainty Modelling in Deep Networks: Forecasting Short and Noisy Series

Axel Brando, José A. Rodríguez-Serrano, Mauricio Ciprian, Roberto Maestre, Jordi Vitrià

TL;DR

This paper introduces a deep learning framework for financial time-series forecasting that doesn't just predict a value, but also estimates its own uncertainty. By modeling "heteroscedastic" aleatoric uncertainty—noise that varies depending on the input—the authors created a system that can proactively discard low-confidence predictions, significantly improving the reliability of financial alerts in a real-world mobile banking app.

What Problem Does This Paper Address?

Forecasting personal financial expenses is notoriously difficult. Unlike physical systems, human spending habits produce "short and noisy" series—often just a few dozen data points that are prone to erratic spikes or sudden changes.

Standard Deep Learning (DL) models for regression typically provide a point estimate: they give you a single number (e.g., "You will spend $50 on coffee next month"). However, they don't tell you how sure they are. In a production environment like a banking app, showing a wildly incorrect prediction is worse than showing nothing at all because it erodes user trust. The challenge is creating a model that can "know when it doesn't know," allowing the system to filter out unreliable forecasts.

Methods

The authors move beyond standard Mean Squared Error (MSE) loss to a probabilistic approach. Instead of predicting a single value $y$, the network predicts the parameters of a probability distribution.

  • Laplace Distribution: While many researchers use a Normal (Gaussian) distribution, this paper opts for a Laplace distribution. They found it more stable during training for noisy data because it is less sensitive to extreme outliers than the squared-error nature of the Gaussian.
  • Heteroscedastic Aleatoric Uncertainty: The core innovation is modeling noise that changes based on the input ($x$). The network has two "heads": one predicts the expected value (the forecast) and the other predicts the "scale" or uncertainty ($b_{ale}$).
  • Architectures: They tested both standard Dense Networks and Long Short-Term Memory (LSTM) networks, which are specifically designed to handle sequential data.
  • Error-Reject Trade-off: They implemented a mechanism to rank predictions by their predicted uncertainty. By setting a threshold, the system can "reject" (hide from the user) the most uncertain forecasts.

Datasets

The study uses a massive, real-world industrial dataset from BBVA:

  • Scale: 2 million anonymized monthly time series for training and 1 million for testing.
  • Structure: Each series consists of 24 months of aggregated financial data (expenses and incomes) per category (e.g., groceries, rent, salary).
  • Preprocessing: Data was normalized using a scale-invariant transformation to focus on patterns rather than absolute currency values, which can vary wildly between users.

Key Insights and Ideas

  • Aleatoric > Epistemic (for this task): The authors found that modeling data noise (Aleatoric) was more effective than modeling model-weight uncertainty (Epistemic, like Dropout-as-Bayesian-Inference). In high-noise, high-data regimes, the inherent "messiness" of the data is a bigger hurdle than the model's lack of knowledge.
  • Uncertainty Improves Accuracy: Interestingly, training the model to estimate variance doesn't just give you an uncertainty score; it actually helps the model learn the mean more accurately. By accounting for noise, the model becomes more robust.
  • The "Predictable" Subset: The "aha moment" is the Error-Keep curve. The results show that if you are willing to only show the top 25% most confident predictions, the error rate (MAE) drops drastically (e.g., from ~218 down to ~5 in the LSTM model). This confirms that some financial behaviors are highly predictable, while others are essentially random.

Key Results

The proposed LSTMHet (LSTM with Heteroscedastic uncertainty) outperformed all baselines, including the Random Forest model previously used in production.

  • Raw Performance: At 100% "keep" rate (predicting everything), the LSTMHet achieved a Mean Absolute Error (MAE) of 218.1, compared to 232.1 for the Random Forest baseline.
  • The Power of Rejection: When the model was allowed to select only the most confident 25% of cases (using the predicted $b_{het}$ score), the MAE plummeted to 5.04.
  • Comparison to Trivial Baselines: Simple heuristics like "predict the mean" or "predict the same as last month" were significantly worse, with MAEs often double or triple those of the deep learning models at various retention rates.

Limitations and Future Work

  • Unknown Unknowns: The authors acknowledge that while the model handles "known noise" well, it still struggles with "unknown unknowns"—out-of-distribution events that the model hasn't seen before.
  • Combining Uncertainties: A suggested next step is combining Aleatoric and Epistemic uncertainty into a single unified model to capture both data noise and model ignorance.
  • Feature Engineering: While the DL models performed well using only raw time-series data, the authors noted that the Random Forest baseline used many more hand-crafted attributes. Integrating these features into the DL architecture could yield even better results.

Summary updated

Abstract

Deep Learning is a consolidated, state-of-the-art Machine Learning tool to fit a function \(y=f(x)\) when provided with large data sets of examples \(\{(x_i, y_i)\}\). However, in regression tasks, the straightforward application of Deep Learning models provides a point estimate of the target. In addition, the model does not take into account the uncertainty of a prediction. This represents a great limitation for tasks where communicating an erroneous prediction carries a risk. In this paper we tackle a real-world problem of forecasting impending financial expenses and incomings of customers, while displaying predictable monetary amounts on a mobile app. In this context, we investigate if we would obtain an advantage by applying Deep Learning models with a Heteroscedastic model of the variance of a network’s output. Experimentally, we achieve a higher accuracy than non-trivial baselines. More importantly, we introduce a mechanism to discard low-confidence predictions, which means that they will not be visible to users. This should help enhance the user experience of our product.
BibTeX
@inproceedings{brando2018uncertainty,
  author = {Axel Brando and José A. Rodríguez-Serrano and Mauricio Ciprian and Roberto Maestre and Jordi Vitrià},
  title = {{Uncertainty Modelling in Deep Networks: Forecasting Short and Noisy Series}},
  booktitle = {ECML/PKDD},
  year = {2018},
  url = {https://openreview.net/forum?id=fUmozAMpcv},
}

Loading related content...

Figure