TL;DR
This paper bridges the gap between deep neural networks (DNNs) and traditional choice modeling by providing a statistical learning theory framework. It demonstrates that DNNs can be both more predictive and interpretable than standard models like binary logit, even in smaller datasets, by focusing on function estimation rather than parameter estimation.
What Problem Does This Paper Address?
Researchers are increasingly using deep neural networks (DNNs) for choice analysis, but two major hurdles remain: overfitting in smaller datasets and a perceived lack of interpretability. Traditional discrete choice models (DCMs) are interpretable but less flexible and predictive. This paper aims to resolve these issues by developing a theoretical framework grounded in statistical learning theory to understand the trade-offs between prediction and interpretation, and between estimation and approximation errors in DNNs for choice analysis.
Methods
The authors propose a statistical learning theory framework that decomposes choice analysis into two dimensions: estimation vs. approximation error, and prediction vs. interpretation loss. They define interpretation loss as the difference between true and estimated choice probability functions, allowing for a direct comparison with prediction loss. The paper then uses this framework to analyze DNNs and binary logit models (BNLs) through theoretical proofs and simulations. Key theoretical contributions include:
- Decomposition of errors: Formalizing the trade-off between estimation and approximation errors in DNNs.
- Quantifying interpretation loss: Defining and measuring interpretation loss based on choice probability functions.
- Theoretical bounds: Using statistical learning theory (Rademacher complexity) to bound the estimation error of DNNs, showing that parameter count is less important than parameter norms.
- BNL as a special case of DNN: Demonstrating that BNL is a subset of DNNs, providing a theoretical basis for comparison.
Datasets
- Simulated Datasets: Three scenarios with synthetic data were generated to simulate typical research situations, varying sample size (100 to 1 million) and number of input variables (20 or 50).
- National Household Travel Survey (NHTS) 2017: Real-world data was used to analyze travel mode and trip purpose choices, with sample sizes ranging from 100 to 500,000.
Key Insights and Ideas
- Interpretation is about Functions, Not Just Parameters: The paper argues that interpretability in DNNs for choice analysis should focus on the estimated choice probability function, not individual parameters, which are often too numerous and complex to interpret. This aligns with extracting economic information from choice models.
- DNNs Can Be Interpretable and Predictive: Contrary to the "black box" perception, DNNs, when analyzed through the lens of choice probability functions, can offer both superior prediction and interpretable insights, especially when traditional models are misspecified.
- Sample Size Matters, But Not Always as Expected: While larger sample sizes generally improve DNN performance, the paper shows that DNNs can outperform BNLs even with moderately sized datasets (around 10^4 samples) when BNLs are misspecified. The theoretical bounds on estimation error are more sensitive to parameter norms and sample size than to the sheer number of parameters.
- BNL is a Simpler DNN: The theoretical framework shows BNL as a special case of DNN, highlighting that DNNs are a more general and powerful class of models.
Key Results
- Scenario 1 (BNL is true DGP): DNN and BNL performed similarly, with DNN's estimation error converging slightly slower. Both required around 10^4 samples for near-optimal performance.
- Scenario 2 (BNL misspecified): DNN significantly outperformed BNL in both prediction and interpretation, even at smaller sample sizes, due to its zero approximation error compared to BNL's large error. DNN required larger sample sizes (10^5-10^6) to converge to its optimum.
- Scenario 3 (Both misspecified): DNN still outperformed BNL, though it also incurred approximation errors. This highlights that DNNs, while powerful, are not immune to issues like omitted variables.
- NHTS Dataset: DNN generally outperformed BNL, especially with larger sample sizes, reinforcing the idea that DNNs can leverage more information from large datasets.
Limitations and Future Work
- Theoretical Development Ongoing: The paper acknowledges that statistical learning theory for non-asymptotic regimes and high-dimensional models is still evolving.
- Clear-Cut Sample Size Rule: Providing a definitive rule for the required sample size for DNNs remains challenging, as it depends on various factors like input dimensions, DNN depth, and regularization.
- Future Research Directions: The authors suggest future work on:
- Effective regularization methods for small samples.
- Relating DNNs to mixed logit models with random coefficients.
- Broadening the concept of interpretability beyond traditional choice models.
- Further investigating the approximation error component of DNNs.
Summary updated