Towards forecast techniques for business analysts of large commercial data sets using matrix factorization methods

Rodrigo Rivera, Ivan Nazarov, Evgeny BurnaevJournal of Physics Conference Series 2018 · DOI

Abstract

This research article suggests that there are significant benefits in exposing demand planners to forecasting methods using matrix completion techniques. This study aims to contribute to a better understanding of the field of forecasting with multivariate time series prediction by focusing on the dimension of large commercial data sets with hierarchies. This research highlights that there has neither been sufficient academic research in this sub-field nor dissemination among practitioners in the business sector. This study seeks to innovate by presenting a matrix completion method for short-term demand forecast of time series data on relevant commercial problems. Albeit computing intensive, this method outperforms the state of the art while remaining accessible to business users. The object of research is matrix completion for time series in a big data context within the industry. The subject of the research is forecasting product demand using techniques for multivariate hierarchical time series prediction that are both precise and accessible to non-technical business experts. Apart from a methodological innovation, this research seeks to introduce practitioners to novel methods for hierarchical multivariate time series prediction. The research outcome is of interest for organizations requiring precise forecasts yet lacking the appropriate human capital to develop them.

Introduction

In the retail sector, "Out-of-Stock" (OOS) events represent a multi-billion dollar challenge, impacting both immediate revenue and long-term customer loyalty. Effective demand forecasting is the primary defense against such inefficiencies. However, a significant "skills gap" exists in the industry. Traditional statistical methods like ARIMA are user-friendly but often fail to capture the complex, hierarchical nature of modern commercial data. Conversely, state-of-the-art machine learning ensembles, while highly accurate, demand extensive feature engineering and specialized data science teams—resources many organizations lack.

This research proposes a "middle way": a matrix factorization-based approach that is sophisticated enough to handle large-scale hierarchical data while remaining automated enough for business analysts to implement without deep technical expertise.

Methodology

The core of the proposed method is Matrix Factorization (MF), which decomposes a large matrix of observations into latent factors (representing temporal patterns) and factor loadings (representing product relationships). To adapt MF for demand forecasting, two critical enhancements are introduced:

  1. Temporal Regularization (TRMF): By imposing an Autoregressive (AR) penalty on latent factors, the model is forced to learn patterns predictive of future states, enabling robust forecasting.
  2. Graph Regularization: This term is applied to factor loadings to incorporate hierarchical structures. It ensures that related products (e.g., items within the same category) share information, which is particularly beneficial for products with sparse sales histories.

Datasets and Experimental Setup

The approach was validated using two distinct datasets representing different commercial challenges: a proprietary manufacturing dataset with a deep hierarchy and the Grupo Bimbo inventory demand dataset, a "flat" but high-volume dataset characterized by very short time series.

Summary of the datasets used in the study, highlighting differences in product count, time periods, and hierarchical structure.

Results and Discussion

The Matrix Completion method demonstrated superior performance in short-term accuracy compared to industry-standard ARIMAX and complex ML Ensembles, as measured by Symmetric Mean Absolute Percent Error (SMAPE).

Comparative performance (SMAPE) of ARIMAX, Ensemble, and Matrix Completion methods across different datasets and time horizons.

Key findings from the experimental results include:

  • Automated Feature Discovery: Unlike ensembles that required over 360 manual features, the Matrix Completion approach reached higher accuracy by automatically identifying latent structures in the data.
  • Hierarchical Stability: Graph regularization prevents erratic forecasts for individual products by aligning them with broader category trends, acting as a stabilizer for the model.
  • Cold Start Capability: The model can address the "cold start" problem for new products by assigning them baseline forecasts derived from the latent factors of their parent categories.

Conclusion

While highly accurate, the proposed method is computationally intensive, requiring significant processing time for large datasets. It also showed performance degradation over longer horizons compared to ensembles. However, the framework offers a powerful tool for organizations needing precise forecasts without the overhead of a large data science department. By bridging the gap between simplicity and performance, it empowers business analysts to leverage big data effectively for short-term demand planning.

Summary updated

Abstract

This research article suggests that there are significant benefits in exposing demand planners to forecasting methods using matrix completion techniques. This study aims to contribute to a better understanding of the field of forecasting with multivariate time series prediction by focusing on the dimension of large commercial data sets with hierarchies. This research highlights that there has neither been sufficient academic research in this sub-field nor dissemination among practitioners in the business sector. This study seeks to innovate by presenting a matrix completion method for short-term demand forecast of time series data on relevant commercial problems. Albeit computing intensive, this method outperforms the state of the art while remaining accessible to business users. The object of research is matrix completion for time series in a big data context within the industry. The subject of the research is forecasting product demand using techniques for multivariate hierarchical time series prediction that are both precise and accessible to non-technical business experts. Apart from a methodological innovation, this research seeks to introduce practitioners to novel methods for hierarchical multivariate time series prediction. The research outcome is of interest for organizations requiring precise forecasts yet lacking the appropriate human capital to develop them.
BibTeX
@article{rivera2018towards,
  author = {Rodrigo Rivera and Ivan Nazarov and Evgeny Burnaev},
  title = {{Towards forecast techniques for business analysts of large commercial data sets using matrix factorization methods}},
  journal = {Journal of Physics Conference Series},
  year = {2018},
  doi = {10.1088/1742-6596/1117/1/012010},
}
Figure