Deep learning in business analytics and operations research: Models, applications and managerial implications

Mathias Kraus, Stefan Feuerriegel, Asil Oztekin

TL;DR

This paper bridges the gap between deep learning's potential and its practical application in business analytics and operations research. It introduces a novel "deep-embedded" network architecture that effectively handles heterogeneous data (combining categorical and numerical features) and demonstrates its superior performance over traditional methods and standard deep learning models across three real-world case studies.

What Problem Does This Paper Address?

Business analytics is increasingly reliant on predictive models to drive value, but traditional machine learning methods are hitting performance plateaus, especially with the advent of "big data." Deep learning, with its ability to model complex, non-linear relationships, offers a promising avenue for improvement. However, research applying deep learning to business analytics and operations research is scarce. Furthermore, a significant challenge in this domain is effectively handling categorical variables, which often require complex preprocessing or lead to numerically unstable models. This paper aims to address this gap by reviewing deep learning concepts, proposing a novel architecture for handling mixed data types, and demonstrating its effectiveness through practical applications.

Methods

The authors propose a deep-embedded network architecture that combines two key components:

  1. Embedding Layers: These layers map categorical variables into dense, low-dimensional vector representations. This is crucial for handling a large number of categories without the numerical instability associated with one-hot encodings.
  2. Stacked Neural Layers: The embedded categorical features are then concatenated with numerical features and fed into standard neural network layers (MLPs, CNNs, or RNNs).

This architecture is trained end-to-end, meaning the embeddings are learned jointly with the rest of the network, which is particularly beneficial when data is scarce. The authors also incorporate dropout and batch normalization to combat overfitting and improve training stability. They compare their proposed architecture against various traditional machine learning models (Lasso, Ridge, Random Forest, SVM, single-layer neural networks) and standard deep learning models (default DNNs, LSTMs).

Datasets

The paper utilizes three real-world datasets for its case studies:

  • Case Study 1 (IT Service Request Load Forecasting): A proprietary dataset from a Swiss organization containing hourly helpdesk ticket data from January 2011 to November 2017. This dataset includes a categorical "holiday flag" and numerical time-series features.
  • Case Study 2 (Sales Forecasting): A public dataset from Kaggle (rossmann-store-sales) comprising daily sales data from 1113 pharmacies. This dataset features categorical variables like store ID, holidays, and store type, along with numerical variables like day of week and week of year.
  • Case Study 3 (Insurance Credit Scoring): A public dataset from a Kaggle competition containing real-world customer records with 43 numerical and 14 categorical covariates for automotive insurance. The prediction task is binary: whether a customer will file a claim.

Key Insights and Ideas

  • Deep Learning's Untapped Potential in OR: Despite the surge in deep learning, its adoption in business analytics and operations research is still nascent, presenting a significant opportunity.
  • Categorical Data is a Bottleneck: Traditional methods struggle with categorical variables, and standard deep learning approaches can lead to numerical instability. The proposed embedding approach directly addresses this.
  • Custom Architectures Outperform Off-the-Shelf: Default, out-of-the-box deep learning architectures are often suboptimal. Tailoring the architecture, like the deep-embedded network, is key to achieving superior performance.
  • Interpretability is Gaining Traction: While deep learning models are often "black boxes," the paper highlights techniques like partial dependence plots and Shapley values, and proposes a novel visualization combining t-SNE with marginal effects for better understanding.
  • Big Data is Crucial: Deep learning models, unlike many traditional methods, continue to benefit from larger datasets, making them ideal for the era of big data.

Key Results

Across all three case studies, the proposed deep-embedded network architecture consistently outperformed both traditional machine learning models and default deep learning architectures.

  • Load Forecasting: Achieved a 45.10% improvement over the mean predictor and a 3.75% improvement over the best baseline (a default DNN), with statistically significant improvements in mean absolute error and substantial cost savings.
  • Sales Forecasting: Showed an impressive improvement of up to 89.42% compared to predicting the mean value, with the deep-embedded LSTM decreasing mean squared error by 21.73% compared to a default DNN.
  • Insurance Credit Scoring: The deep-embedded DNN improved AUC by 1.56% and Gini coefficient by 7.14% over the best traditional model (ridge regression), resulting in significantly lower misclassification costs.

The paper also notes that while deep neural networks require more computational time for optimization, this is often countervailed by their higher prediction performance.

Limitations and Future Work

  • Parameter Tuning: Optimizing deep neural networks is challenging and requires extensive parameter tuning. The authors mention AutoML as a promising direction for automating this process.
  • Uncertainty Quantification: Current DNNs primarily provide point estimates, lacking rigorous uncertainty quantification. Future work could focus on developing methods for modeling distributions or Bayesian variants of DNNs.
  • Interpretability and Accountability: While progress has been made, these remain weaknesses in deep learning. Future research could explore estimating structural models via variational inference or interpreting attention mechanisms.
  • Translating Predictions to Decisions: Deep learning provides insights but not direct management strategies. Future studies could focus on how to translate predictions into effective decision-making, potentially using techniques like inverse control or Markov decision processes.
  • Sequence-to-Sequence Models: The paper notes that current sequence learning models still output fixed-dimension vectors. Exploring "seq2seq" techniques for generating variable-length sequences (like routes or graphs) is a promising avenue.
  • Transfer Learning: For domains with limited data, transfer learning could be a valuable strategy to leverage existing knowledge.
  • Generative Adversarial Networks (GANs): While GANs have shown promise in other fields, their application and benefit in operations research are yet to be fully demonstrated.

Summary updated

Abstract

Business analytics refers to methods and practices that create value through data for individuals, firms, and organizations. This field is currently experiencing a radical shift due to the advent of deep learning: deep neural networks promise improvements in prediction performance as compared to models from traditional machine learning. However, our research into the existing body of literature reveals a scarcity of research works utilizing deep learning in our discipline. Accordingly, the objectives of this overview article are as follows: (1) we review research on deep learning for business analytics from an operational point of view. (2) We motivate why researchers and practitioners from business analytics should utilize deep neural networks and review potential use cases, necessary requirements, and benefits. (3) We investigate the added value to operations research in different case studies with real data from entrepreneurial undertakings. All such cases demonstrate improvements in operational performance over traditional machine learning and thus direct value gains. (4) We provide guidelines and implications for researchers, managers and practitioners in operations research who want to advance their capabilities for business analytics with regard to deep learning. (5) Our computational experiments find that default, out-of-the-box architectures are often suboptimal and thus highlight the value of customized architectures by proposing a novel deep-embedded network.
BibTeX
@article{kraus2020deep,
  author = {Mathias Kraus and Stefan Feuerriegel and Asil Oztekin},
  title = {{Deep learning in business analytics and operations research: Models, applications and managerial implications}},
  journal = {Eur. J. Oper. Res.},
  year = {2020},
  doi = {10.1016/j.ejor.2019.09.018},
}
Figure