Exploring the representativeness of the M5 competition data

Evangelos Theodorou, Shengjie Wang, Yanfei Kang, Evangelos Spiliotis, Spyros Makridakis, Vassilios AssimakopoulosarXiv 2021 · arXiv

Abstract

The M5 forecasting competition, based on Walmart sales data, has significantly influenced the field of time series analysis. However, questions remain regarding the generalizability of its findings to other retail contexts. This paper investigates whether the M5 "feature space" is representative of broader retail industries by comparing it against datasets from major retailers in Ecuador and Greece. Using feature-based instance space analysis, the authors show that M5 data effectively covers the patterns found in these diverse environments, suggesting that the competition's winning strategies have broad applicability.

Introduction: The Generalizability Challenge

Forecasting competitions like the M5 serve as benchmarks for machine learning methods in time series analysis. Despite their success, practitioners often question the "particularity" of the underlying data. If a competition relies solely on Walmart's US sales data, can its results be trusted by a grocery chain in Europe or South America? Differences in regional consumer behavior, product categories, and marketing strategies—such as Walmart’s "Everyday Low Price" model versus heavy promotional discounting—might create fundamentally different data patterns. This study quantifies the representativeness of the M5 data to determine if its "best practices" are truly universal.

Methodology: Feature-Based Instance Space Analysis

The authors move beyond raw sales comparisons to analyze the mathematical "fingerprints" or features of the time series.

  1. Feature Extraction: A pool of 2,508 features was generated using tsfresh and tsfeatures across daily, weekly, and monthly scales.
  2. Feature Selection: To manage dimensionality, the authors used a pipeline involving Kruskal-Wallis tests for class distinction, the RReliefF algorithm for forecasting relevance, and hierarchical clustering to minimize redundancy. This resulted in a refined set of 42 key features, including entropy, trend strength, and intermittency.
  3. Visualization: Using t-SNE (t-Stochastic Neighbor Embedding), these 42 dimensions were projected onto a 2D map to measure "miscoverage" and "non-overlapping ratios" between datasets.

Datasets and Comparative Analysis

The study compares three distinct retail environments to test the breadth of the M5 data.

Aggregated time series comparison across the three datasets: M5 (Walmart), Greek retail firm, and Corporación Favorita.
  • M5 (Walmart): Comprising 30,490 series from the US (2011–2016), this dataset is known for high intermittency.
  • Corporación Favorita: A massive dataset of 174,654 series from a leading Ecuadorian retailer (2013–2017).
  • Greek Supermarket Chain: 7,248 series from Greece (2018–2020), notably covering the COVID-19 lockdown period.

Key Insights: The Universality of Retail Patterns

The analysis revealed that while the M5 data is significantly more intermittent than the others, its overall diversity makes it a robust "superset" of retail patterns.

Summary statistics and time series characteristics for the three datasets.
  • Intermittency and Diversity: M5 features a high percentage of intermittent series (73.35%), yet it successfully encompasses the "lumpy" and "erratic" patterns found in the Greek and Ecuadorian data.
  • Strategy Neutrality: Despite differing marketing strategies (constant low prices vs. heavy promotions), the resulting sales features occupy similar regions in the instance space.
  • Pandemic Resilience: The Greek data, which includes extreme structural breaks from COVID-19 lockdowns, still fell within the feature space defined by the M5 data, indicating the robustness of retail sales characteristics.

Quantitative Results

The representativeness of M5 is supported by low miscoverage rates:

  • Miscoverage: Only 0.67% of the Greek feature space and 5.56% of the larger Ecuadorian feature space were unrepresented in the M5 data.
  • Non-Overlapping Ratio (NOR): Only 0.37% of the Greek time series were located in areas of the feature space not covered by M5.

Conclusion and Limitations

While the M5 data is remarkably representative of brick-and-mortar grocery retail, the study notes that online-only retailers or specialized sectors (e.g., pharmaceuticals) might exhibit different patterns. Additionally, the analysis focuses on the sales signal itself rather than explanatory variables like weather or specific promotional calendars. Nonetheless, the findings provide strong evidence that the M5 competition offers valuable, generalizable insights for the broader retail industry.

Summary updated

BibTeX
@misc{theodorou2021exploring,
  author = {Evangelos Theodorou and Shengjie Wang and Yanfei Kang and Evangelos Spiliotis and Spyros Makridakis and Vassilios Assimakopoulos},
  title = {{Exploring the representativeness of the M5 competition data}},
  year = {2021},
  eprint = {2103.02941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.02941},
}
Figure