Deep Neural Networks for Choice Analysis: A Statistical Learning Theory Perspective

Shenhao Wang, Qingyi Wang, Nate Bailey, Jinhua ZhaoTransportation 2018 · arXiv

TL;DR

This paper bridges the gap between deep neural networks (DNNs) and traditional choice modeling by providing a statistical learning theory framework. It demonstrates that DNNs can be both more predictive and interpretable than standard models like binary logit, even in smaller datasets, by focusing on function estimation rather than parameter estimation.

What Problem Does This Paper Address?

Researchers are increasingly using deep neural networks (DNNs) for choice analysis, but two major hurdles remain: overfitting in smaller datasets and a perceived lack of interpretability. Traditional discrete choice models (DCMs) are interpretable but less flexible and predictive. This paper aims to resolve these issues by developing a theoretical framework grounded in statistical learning theory to understand the trade-offs between prediction and interpretation, and between estimation and approximation errors in DNNs for choice analysis.

Methods

The authors propose a statistical learning theory framework that decomposes choice analysis into two dimensions: estimation vs. approximation error, and prediction vs. interpretation loss. They define interpretation loss as the difference between true and estimated choice probability functions, allowing for a direct comparison with prediction loss. The paper then uses this framework to analyze DNNs and binary logit models (BNLs) through theoretical proofs and simulations. Key theoretical contributions include:

  • Decomposition of errors: Formalizing the trade-off between estimation and approximation errors in DNNs.
  • Quantifying interpretation loss: Defining and measuring interpretation loss based on choice probability functions.
  • Theoretical bounds: Using statistical learning theory (Rademacher complexity) to bound the estimation error of DNNs, showing that parameter count is less important than parameter norms.
  • BNL as a special case of DNN: Demonstrating that BNL is a subset of DNNs, providing a theoretical basis for comparison.

Datasets

  • Simulated Datasets: Three scenarios with synthetic data were generated to simulate typical research situations, varying sample size (100 to 1 million) and number of input variables (20 or 50).
  • National Household Travel Survey (NHTS) 2017: Real-world data was used to analyze travel mode and trip purpose choices, with sample sizes ranging from 100 to 500,000.

Key Insights and Ideas

  • Interpretation is about Functions, Not Just Parameters: The paper argues that interpretability in DNNs for choice analysis should focus on the estimated choice probability function, not individual parameters, which are often too numerous and complex to interpret. This aligns with extracting economic information from choice models.
  • DNNs Can Be Interpretable and Predictive: Contrary to the "black box" perception, DNNs, when analyzed through the lens of choice probability functions, can offer both superior prediction and interpretable insights, especially when traditional models are misspecified.
  • Sample Size Matters, But Not Always as Expected: While larger sample sizes generally improve DNN performance, the paper shows that DNNs can outperform BNLs even with moderately sized datasets (around 10^4 samples) when BNLs are misspecified. The theoretical bounds on estimation error are more sensitive to parameter norms and sample size than to the sheer number of parameters.
  • BNL is a Simpler DNN: The theoretical framework shows BNL as a special case of DNN, highlighting that DNNs are a more general and powerful class of models.

Key Results

  • Scenario 1 (BNL is true DGP): DNN and BNL performed similarly, with DNN's estimation error converging slightly slower. Both required around 10^4 samples for near-optimal performance.
  • Scenario 2 (BNL misspecified): DNN significantly outperformed BNL in both prediction and interpretation, even at smaller sample sizes, due to its zero approximation error compared to BNL's large error. DNN required larger sample sizes (10^5-10^6) to converge to its optimum.
  • Scenario 3 (Both misspecified): DNN still outperformed BNL, though it also incurred approximation errors. This highlights that DNNs, while powerful, are not immune to issues like omitted variables.
  • NHTS Dataset: DNN generally outperformed BNL, especially with larger sample sizes, reinforcing the idea that DNNs can leverage more information from large datasets.

Limitations and Future Work

  • Theoretical Development Ongoing: The paper acknowledges that statistical learning theory for non-asymptotic regimes and high-dimensional models is still evolving.
  • Clear-Cut Sample Size Rule: Providing a definitive rule for the required sample size for DNNs remains challenging, as it depends on various factors like input dimensions, DNN depth, and regularization.
  • Future Research Directions: The authors suggest future work on:
  • Effective regularization methods for small samples.
  • Relating DNNs to mixed logit models with random coefficients.
  • Broadening the concept of interpretability beyond traditional choice models.
  • Further investigating the approximation error component of DNNs.

Summary updated

Abstract

While researchers increasingly use deep neural networks (DNN) to analyze individual choices, overfitting and interpretability issues remain as obstacles in theory and practice. By using statistical learning theory, this study presents a framework to examine the tradeoff between estimation and approximation errors, and between prediction and interpretation losses. It operationalizes the DNN interpretability in the choice analysis by formulating the metrics of interpretation loss as the difference between true and estimated choice probability functions. This study also uses the statistical learning theory to upper bound the estimation error of both prediction and interpretation losses in DNN, shedding light on why DNN does not have the overfitting issue. Three scenarios are then simulated to compare DNN to binary logit model (BNL). We found that DNN outperforms BNL in terms of both prediction and interpretation for most of the scenarios, and larger sample size unleashes the predictive power of DNN but not BNL. DNN is also used to analyze the choice of trip purposes and travel modes based on the National Household Travel Survey 2017 (NHTS2017) dataset. These experiments indicate that DNN can be used for choice analysis beyond the current practice of demand forecasting because it has the inherent utility interpretation, the flexibility of accommodating various information formats, and the power of automatically learning utility specification. DNN is both more predictive and interpretable than BNL unless the modelers have complete knowledge about the choice task, and the sample size is small. Overall, statistical learning theory can be a foundation for future studies in the non-asymptotic data regime or using high-dimensional statistical models in choice analysis, and the experiments show the feasibility and effectiveness of DNN for its wide applications to policy and behavioral analysis.
BibTeX
@inproceedings{wang2018deep,
  author = {Shenhao Wang and Qingyi Wang and Nate Bailey and Jinhua Zhao},
  title = {{Deep Neural Networks for Choice Analysis: A Statistical Learning Theory Perspective}},
  booktitle = {Transportation},
  year = {2018},
}
Figure