Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Data

Hanqi Su, Binyang Song, Faez Ahmed

Introduction

Vehicle evaluation is a high-stakes process for both manufacturers and consumers. While humans naturally synthesize diverse information—looking at a car's silhouette, reading its technical specifications, and scanning expert reviews—most existing machine learning models are "unimodal," focusing on a single data type. This research from MIT’s Department of Mechanical Engineering addresses this gap by developing a multi-modal approach that "sees" a vehicle more like a human consumer does, leading to more accurate predictions and actionable design insights.

Methodology

The researchers developed a framework that simultaneously learns from three distinct data "modes" to predict five key scores: Total, Critics, Performance, Safety, and Interior. The architecture consists of three specialized unimodal branches that feed into a central multi-modal model.

Overview of the multi-modal learning framework for vehicle rating prediction.
  1. Parametric Branch: Processes 302 technical features (e.g., engine capacity, MSRP, dimensions) using a Multi-Layer Perceptron (MLP).
  2. Text Branch: Encodes textual descriptions and reviews using a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model.
  3. Image Branch: Utilizes a VGG16 CNN architecture with an added self-attention mechanism. This allows the model to weigh the importance of different regions in composite images, which include four exterior and four interior views.

The final latent representations (embeddings) from each branch are concatenated—a strategy known as "late fusion"—and passed through a final dense layer to predict continuous rating scores.

Dataset

The study utilized a novel multi-modal dataset scraped from U.S. News, comprising 2,571 unique vehicle models from 2007 to 2022. Each entry includes 302 parametric features, a text summary of pros and cons, and eight photos. The models were trained to predict five distinct expert-assigned rating scores on a 0–10 scale.

Key Insights

  • Modality Strength: Technical specifications (parametric data) provide the strongest individual signal for predicting ratings. However, they are insufficient to capture the full consumer experience.
  • Synergy through Fusion: The multi-modal model significantly outperforms unimodal models, particularly for the "Total Score," which represents a complex synthesis of aesthetics, specifications, and performance.
  • Feature Evolution: Using SHAP (Shapley Additive exPlanations), the authors tracked how feature importance changed over time. For instance, "Back-up Cameras" became increasingly influential over the last decade, while "HID Headlights" lost favor as LED technology became the new standard.
  • Visual Attention: For interior ratings, the model's attention focused primarily on the dashboard and steering wheel, confirming these as critical touchpoints for perceived quality.

Results

The multi-modal approach demonstrated a clear performance boost, with explanatory power ($R^2$) increasing by 4% to 12% over the best unimodal models.

Comparison of R-squared values across different rating categories for unimodal models.

Specifically, the multi-modal model achieved an $R^2$ of 0.74 for the Total Score, outperforming the parametric-only model (0.62) and the image-only model (0.46). Safety predictions were particularly strong in the parametric model ($R^2 = 0.82$), likely due to the direct correlation between safety ratings and the specific safety technologies listed in technical specifications.

Limitations and Future Work

While promising, the research notes several areas for growth. The dataset size (~2,500 samples) is relatively small for deep learning, potentially leading to overfitting. The authors also suggest that more advanced attention-based fusion techniques could improve upon the current concatenation method. Finally, expanding the dataset beyond the U.S. market would help account for differing global consumer preferences.

Summary updated

Abstract

Abstract Accurate vehicle rating prediction can facilitate designing and configuring good vehicles. This prediction allows vehicle designers and manufacturers to optimize and improve their designs in a timely manner, enhance their product performance, and effectively attract consumers. However, most of the existing data-driven methods rely on data from a single mode, e.g., text, image, or parametric data, which results in a limited and incomplete exploration of the available information. These methods lack comprehensive analyses and exploration of data from multiple modes, which probably leads to inaccurate conclusions and hinders progress in this field. To overcome this limitation, we propose a multi-modal learning model for more comprehensive and accurate vehicle rating predictions. Specifically, the model simultaneously learns features from the parametric specifications, text descriptions, and images of vehicles to predict five vehicle rating scores, including the total score, critics score, performance score, safety score, and interior score. We compare the multi-modal learning model to the corresponding unimodal models and find that the multi-modal model’s explanatory power is 4%–12% higher than that of the unimodal models. On this basis, we conduct sensitivity analyses using SHAP to interpret our model and provide design and optimization directions to designers and manufacturers. Our study underscores the importance of the data-driven multi-modal learning approach for vehicle design, evaluation, and optimization. We have made the code publicly available at http://decode.mit.edu/projects/vehicleratings/.
BibTeX
@misc{su2023multi,
  author = {Hanqi Su and Binyang Song and Faez Ahmed},
  title = {{Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Data}},
  year = {2023},
  eprint = {2305.15218},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.15218},
}
Figure