TL;DR
This paper introduces ASU-DNN, a novel deep neural network architecture for choice analysis that incorporates domain-specific behavioral knowledge (alternative-specific utility functions). This design leads to improved prediction accuracy and interpretability compared to standard fully connected DNNs, demonstrating that integrating behavioral theory can enhance machine learning models.
What Problem Does This Paper Address?
Choice analysis, a field crucial for economics, transportation, and marketing, traditionally relies on discrete choice models. While deep neural networks (DNNs) offer powerful predictive capabilities, their application in choice analysis faces three main challenges:
- Reconciling domain knowledge with generic DNNs: How to effectively integrate specific behavioral insights into general-purpose DNNs.
- Interpretability: DNNs are often "black boxes," making it difficult to extract policy or behavioral insights, unlike traditional choice models.
- Effective regularization and architecture design: Designing DNN architectures and regularization methods that improve out-of-sample performance without systematic guidance.
This paper aims to address these challenges by proposing a DNN architecture that is guided by behavioral knowledge.
Methods
The core innovation is the Alternative-Specific Utility Deep Neural Network (ASU-DNN). Unlike a fully connected DNN (F-DNN) where an alternative's utility depends on all alternatives' attributes, ASU-DNN models the utility of an alternative k based solely on its own attributes. This design is inspired by the concept of alternative-specific utility functions in traditional choice models.
Key aspects of the method include:
- Architecture Design: ASU-DNN is structured as a stack of subnetworks, each computing a utility score for an alternative, with individual-specific variables interacting after initial layers. This results in a sparser connectivity compared to F-DNN.
- Domain-Knowledge-Based Regularization: The alternative-specific connectivity inherently enforces an Independence of Irrelevant Alternatives (IIA) constraint, which acts as a form of domain-specific regularization. This is argued to be more effective than generic regularization methods.
- Theoretical Analysis: The paper uses statistical learning theory to analyze the trade-off between estimation and approximation errors for ASU-DNN and F-DNN, suggesting ASU-DNN has lower estimation error due to its sparsity.
- Behavioral Test: The comparison between ASU-DNN and F-DNN serves as a test of behavioral hypotheses, investigating whether individuals compute utility based on an alternative's own attributes or those of all alternatives.
Datasets
The study uses two datasets:
- SGP Dataset: An online survey collected in Singapore in 2017, containing travel mode choices (walking, public transit, driving, ride sharing, autonomous vehicles) and socioeconomic variables. It has 8,418 observations.
- TRAIN Dataset: A public dataset from the R
mlogitpackage, involving binary travel mode choices (two types of trains) with attributes like price, time cost, and comfort. It has 2,929 observations.
Both datasets were split into training, validation, and testing sets in a 4:1:1 ratio, with five-fold cross-validation used for model selection.
Key Insights and Ideas
- Behavioral Knowledge as a Design Principle: The paper demonstrates that domain-specific behavioral insights (like alternative-specific utility) can effectively guide the design of DNN architectures, leading to improved performance.
- ASU-DNN as a Compromise: ASU-DNN strikes a balance between the automatic feature learning of generic DNNs and the structured knowledge of traditional choice models.
- IIA as Effective Regularization: The implicit IIA constraint in ASU-DNN acts as a powerful domain-knowledge-based regularization, outperforming explicit and implicit generic regularization methods.
- Interpretability through Structure: The structured nature of ASU-DNN leads to more interpretable choice probability functions and elasticity coefficients compared to F-DNN.
- Support for Traditional Choice Modeling: The results empirically support the long-standing assumption in choice modeling that individuals tend to compute utility based on an alternative's own attributes.
Key Results
- Prediction Accuracy: ASU-DNN consistently outperformed F-DNN by approximately 2-3% in prediction accuracy across both datasets and various hyperparameter settings. The top 10 ASU-DNN models showed a significant advantage over the top 10 F-DNN models.
- Comparison with Other Classifiers: Both F-DNN and ASU-DNN significantly outperformed other machine learning classifiers (MNL, NL, SVM, DT, etc.) by about 8% in prediction accuracy.
- Regularization Effectiveness: The alternative-specific connectivity hyperparameter was found to be more effective than explicit regularizations (l1, l2 penalties) and implicit regularizations (learning rate, iterations, batch size, batch normalization) in improving prediction accuracy.
- Interpretability: Visualizations of choice probability functions showed that ASU-DNN's substitution patterns were more intuitive and reasonable than F-DNN's, aligning better with behavioral expectations.
Limitations and Future Work
- Generalizability: While ASU-DNN is guaranteed to have an IIA-constraint substitution pattern and smaller estimation errors than F-DNN, it's not guaranteed to always outperform F-DNN or MNL in prediction accuracy. Performance depends on sample size, model complexity, and the true data generating process (DGP). ASU-DNN is expected to perform better when sample size is smaller, the DGP is closer to IIA, and the choice set is larger.
- Realism of Behavioral Patterns: The paper acknowledges that definitively proving the realism of the behavioral patterns revealed by ASU-DNN is challenging, as the true DGP is unknown. Future studies using simulations are suggested to address this.
- Incorrect Prior Knowledge: The alternative-specific utility specification in ASU-DNN could be incorrect, similar to any modeling practice. However, ASU-DNN's "downward-built" approach from a universal approximator is considered more conservative and robust to misspecification than traditional "upward-built" models.
- Future Directions:
- Investigate the generalizability of ASU-DNN to other contexts.
- Explore using more flexible utility functions (like those in NL and MXL) to design novel DNN architectures.
- Identify regularization methods that correspond to NL or MXL models.
- Use simulations to determine conditions under which ASU-DNN approximates the true DGP more efficiently than F-DNN and MNL.
Summary updated