Images in Discrete Choice Modeling: Addressing Data Isomorphism in Multi-Modality Inputs

Brian Sifringer, Alexandre AlahiarXiv 2023 · DOI · arXiv

Introduction

Discrete Choice Models (DCMs) are fundamental tools in econometrics and transportation science, prized for their ability to provide interpretable parameters ($\beta$ weights) that quantify decision-making factors. However, traditional DCMs are limited to structured tabular data. To incorporate unstructured data like images, researchers have turned to hybrid models that integrate Neural Networks (NNs) into the DCM framework. While these models improve predictive accuracy, they often compromise the interpretability that makes DCMs valuable.

The Problem of Data Isomorphism

A significant challenge in multi-modal choice modeling is "data isomorphism"—the presence of redundant information across different input modalities. For instance, if a vacation rental model includes a tabular variable for "number of bathrooms" and an image clearly showing those bathrooms, a "greedy" Neural Network will often absorb the signal from the image to minimize prediction error. This results in the DCM's $\beta$ weight for the tabular variable becoming biased or dropping toward zero, rendering the model's parameters untrustworthy for policy analysis or behavioral insight.

Conceptual framework of data isomorphism in hybrid choice models: (A) Underspecified model, (B) Biased model with overlapping information, and (C) Optimal model with complementary information.

Methodology

The researchers investigated the Learning Multinomial Logit (L-MNL) framework, which augments a standard logit utility function with a latent representation derived from images via a ResNet18 backbone. To address the isomorphism problem, they evaluated three distinct strategies:

  1. Architectural Adjustments: Utilizing Latent Models and $\beta$-Variational Autoencoders ($\beta$-VAEs) to disentangle information and force the NN to capture only complementary data.
  2. Source Information Masking: Identifying redundant features in images using detection models and obscuring them with black masks.
  3. Background Inpainting: Erasing redundant objects and filling the gaps with background textures to remove the information while minimizing artificial noise.

The study utilized a synthetic reconstruction of the MIT Moral Machine dataset. By generating images from known tabular variables, the authors established a "ground truth" to precisely measure how much information was leaked to the NN and how it affected parameter recovery.

Key Results and Insights

The study revealed that Neural Networks are highly efficient at exploiting any available signal in high-dimensional pixel data, even when that signal is explicitly provided in structured tables. Surprisingly, sophisticated architectures like $\beta$-VAEs failed to prevent parameter bias, suggesting that architectural fixes alone are insufficient.

The most effective solution was found at the data source. By ensuring that the image and tabular data are strictly complementary, model integrity was restored. Specifically:

  • Interpretability Error ($E_\beta$): Naive hybrid models with overlapping data showed a high parameter error of 2.89.
  • The Power of Inpainting: By using 100% scale inpainting to remove redundant objects from the images, the researchers reduced the parameter error to 0.18—almost perfectly recovering the true statistical weights of the DCM.
  • Accuracy vs. Insight: While the hybrid models always had higher predictive accuracy than a simple DCM (approx. 79% vs 77%), the "naive" versions provided no insight into why the accuracy improved. Only the "inpainted" models provided both high accuracy and trustworthy parameters.

Conclusion and Future Directions

Preserving interpretability in hybrid choice models requires rigorous management of data redundancy. The findings suggest that "cleaning" image data to remove information already present in tabular form is essential for valid behavioral modeling.

Future research should focus on applying these techniques to real-world datasets where masking is more complex. Developing robust, saliency-based masking methods and exploring probabilistic congruence—where information overlap is inconsistent—remain key areas for advancing multi-modal discrete choice modeling.

Summary updated

Abstract

This paper explores the intersection of Discrete Choice Modeling (DCM) and machine learning, focusing on the integration of image data into DCM's utility functions and its impact on model interpretability. We investigate the consequences of embedding high-dimensional image data that shares isomorphic information with traditional tabular inputs within a DCM framework. Our study reveals that neural network (NN) components learn and replicate tabular variable representations from images when co-occurrences exist, thereby compromising the interpretability of DCM parameters. We propose and benchmark two methodologies to address this challenge: architectural design adjustments to segregate redundant information, and isomorphic information mitigation through source information masking and inpainting. Our experiments, conducted on a semi-synthetic dataset, demonstrate that while architectural modifications prove inconclusive, direct mitigation at the data source shows to be a more effective strategy in maintaining the integrity of DCM's interpretable parameters. The paper concludes with insights into the applicability of our findings in real-world settings and discusses the implications for future research in hybrid modeling that combines complex data modalities. Full control of tabular and image data congruence is attained by using the MIT moral machine dataset, and both inputs are merged into a choice model by deploying the Learning Multinomial Logit (L-MNL) framework.
BibTeX
@misc{sifringer2023images,
  author = {Brian Sifringer and Alexandre Alahi},
  title = {{Images in Discrete Choice Modeling: Addressing Data Isomorphism in Multi-Modality Inputs}},
  year = {2023},
  eprint = {2312.14724},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.14724},
}
Figure