TL;DR
This paper tackles the critical challenge of making industrial neural forecasting systems robust to real-world data issues like distribution shifts, missing values, and anomalies. They propose an automated system that achieves competitive performance against human-assisted forecasts by employing periodic retraining, native handling of missing data, and a novel anomaly detection mechanism.
What Problem Does This Paper Address?
Deploying machine learning models in production is a significant undertaking, but the real challenges often emerge after deployment. Industrial systems are constantly fed real-world data, which is rarely pristine. This paper focuses on three common data challenges that can degrade the performance of forecasting systems:
- Data Distribution Shifts (Concept Drift): The underlying patterns in the data can change over time due to various factors (e.g., seasonality, business changes), making a model trained on old data less accurate.
- Missing Values: Incomplete data is a frequent occurrence, and many algorithms struggle to handle it gracefully, often requiring complex imputation strategies.
- Anomalies: Outlier data points can arise from events like natural disasters or system failures, and if not handled, they can severely skew forecasts.
The authors aim to build a "resilient" forecasting system that can maintain its predictive performance despite these data imperfections, without constant manual intervention.
Methods
The paper proposes a multi-pronged approach to build a resilient neural forecasting system, primarily using Amazon's DeepAR model for attendance forecasting:
- Periodic Retraining for Distribution Shifts: To combat concept drift, the system is retrained weekly. While neural networks can be more stable than traditional models, they found that weekly retraining significantly improves forecast accuracy (3.9% WMAE improvement). This highlights that data distribution changes are the norm, not the exception, in their industrial setting.
- Native Handling of Missing Values: Instead of relying on external imputation techniques, DeepAR is leveraged for its built-in ability to handle missing values. It does this by masking the loss calculation for missing data points, using indicator features, and automatically imputing values during training. This is a key advantage as it avoids the potential pitfalls of manual imputation, which can introduce biases.
- Anomaly Detection and Mitigation: The system compares incoming actual data with the previous week's forecast. If the difference exceeds a predefined threshold (Δ), the data point is flagged as an anomaly. These anomalous values are then treated as missing values, allowing DeepAR's native handling mechanism to mitigate their impact. This approach is fully automated and scalable.
- Model Stability: To ensure that retraining doesn't introduce excessive variability in forecasts, the authors employ a form of parameter averaging (inspired by Polyak-Ruppert averaging). This technique averages the parameters of the N best models from the training process, significantly improving model stability (42.2% improvement in forecast spread).
Datasets
The paper uses internal Amazon data from fulfillment centers for labor planning, specifically for attendance forecasting. No new benchmark datasets are introduced.
Key Insights and Ideas
- Real-world data is messy, and resilience is key: The paper strongly emphasizes that data quality issues are not edge cases but common challenges in industrial ML. Building systems that can gracefully handle these issues is paramount for sustained performance.
- Native handling of missing values is powerful: Leveraging the inherent capabilities of models like DeepAR to manage missing data simplifies the pipeline and avoids potential biases introduced by external imputation methods.
- Anomaly detection as a precursor to missing value handling: By framing anomaly detection as a way to generate "missing" values, the system effectively reuses existing robust mechanisms, creating a streamlined solution.
- Model stability is crucial for retraining: When dealing with frequent retraining, ensuring that the model's output doesn't fluctuate wildly due to random initialization is critical for business adoption. Model averaging proves effective here.
- Automated systems can rival human expertise: The claim that their fully automated system performs comparably to or better than human-assisted forecasts is a significant finding, suggesting potential for increased efficiency and reduced reliance on manual oversight.
Key Results
- The proposed resilient system performs favorably compared to a hybrid system involving human overrides.
- Weekly retraining improves forecast accuracy by 3.9% (WMAE).
- Model averaging improves model stability by 42.2% (measured by the spread of forecasted values over multiple runs).
- The anomaly detection system, when applied to a specific incident (a week-long fulfillment center closure), reduced the Mean Absolute Error (MAE) from 0.19 to 0.04.
Limitations and Future Work
- Distinguishing anomalies from change points: A key limitation is that the anomaly detection system might incorrectly filter genuine change points (qualitative shifts in the time series), preventing the model from learning the new behavior. The authors address this with a heuristic: if an anomaly is too long, it's treated as a change point. However, this still requires manual flagging for long anomalies.
- Radical data distribution changes: The paper acknowledges that extreme events like the COVID-19 pandemic violate the core assumption that the past predicts the future, rendering many forecasting models temporarily useless. Subject matter experts excel in such situations.
- Future work: The authors suggest that factoring out the impact of major events like pandemics from historical data for future training will be a significant challenge. Further research into safely distinguishing anomalies from change points is also implied.
Summary updated