Generating Vehicular Icon Descriptions and Indications Using Large Vision-Language Models

James Fletcher, Nicholas Dehnen, Seyed Nima Tayarani Bathaie, Aijun An, Heidar Davoudi, Ron DiCarlantonio, Gary FarmanerEMNLP 2024

TL;DR

This paper introduces a novel approach to automatically generate descriptive text for vehicle dashboard icons, enabling better understanding for drivers and enhancing in-car virtual assistants. They developed a new dataset of over 400 icons and found that advanced large vision-language models like GPT-40 and Claude 3.5 perform well, while LLaVA struggles with this specific task.

What Problem Does This Paper Address?

Modern vehicles are equipped with numerous dashboard icons that convey critical information to drivers. However, many drivers are unfamiliar with these symbols, leading to potential confusion and delayed reactions. Existing in-car virtual assistants, while capable of answering questions based on text documents like car manuals, often fail to address icon-related queries due to a lack of icon-specific knowledge. This paper aims to bridge this gap by automatically generating both visual and functional descriptions for these icons, thereby enriching the knowledge base of these assistants and improving driver safety and experience.

Methods

The core of this work involves leveraging large vision-language models (LVLMs) to generate descriptions for vehicle dashboard icons. The authors experimented with three state-of-the-art LVLMs: GPT-40, LLaVA-NEXT, and Claude 3.5. They employed both zero-shot and few-shot prompting techniques to guide the models. For few-shot prompting, they used a method based on the Hamming distance of image hashes to select the most similar training examples. The generated descriptions were then evaluated using a combination of automatic metrics (like BERT-Score, SBERT-Score, CLIP-Score, ROUGE, METEOR) and human evaluation. A key methodological decision was to prompt the models to generate both visual and functional descriptions together, which were then separated, as it proved difficult to prevent models from including functional information when only visual descriptions were requested.

Datasets

The authors created a new dataset specifically for this task, comprising 408 unique vehicle dashboard icon images. These icons were collected from 42 vehicle manuals across four different manufacturers. To ensure diversity and remove exact duplicates while preserving variants, they used dHash image hashing. For each icon, they generated two types of ground-truth descriptions:

  • Visual Description: A description of the icon's visual components. Multiple human annotators contributed to this to capture diverse perspectives.
  • Functional Description: An explanation of the icon's meaning and purpose, derived from the associated car manual text. This was done by native English speakers.

A subset of 20 examples was used for few-shot prompting, and the remaining 388 formed the test set.

Key Insights and Ideas

  • Icons are a Unique Challenge: Unlike natural images, dashboard icons are drawings and often require contextual information from manuals to fully understand their function, making them a distinct challenge for standard image captioning models.
  • Visual vs. Functional Descriptions Matter: Separating visual descriptions (what it looks like) from functional descriptions (what it means) is crucial for building effective question-answering systems, as they form the basis of question-answer pairs.
  • LVLM Performance Varies Significantly: Not all advanced LVLMs are equally adept at this task. GPT-40 and Claude 3.5 showed strong performance, while LLaVA struggled, often exhibiting hallucinations.
  • Context is King (Especially for Function): Functional descriptions benefit significantly from the accompanying text from vehicle manuals, highlighting the importance of multi-modal input.
  • SBERT-Score as a Reliable Metric: Among the automated metrics, SBERT-Score showed the strongest correlation with human judgments, offering a good balance between interpretability and accuracy.

Key Results

  • Top Performers: GPT-40 and Claude 3.5 demonstrated strong performance in generating both visual and functional descriptions, with GPT-40 slightly edging out Claude 3.5 in most metrics.
  • LLaVA's Struggles: LLaVA performed significantly worse than the other two models, particularly in generating accurate descriptions, often resorting to hallucinations.
  • Few-Shot Learning Benefits: For GPT-40 and Claude 3.5, few-shot prompting generally improved performance, with Claude 3.5 showing the most significant gains with 5-shot examples. LLaVA did not show consistent benefits from few-shot prompting.
  • Functional Descriptions Outperform Visual: Across all models, functional descriptions generally scored higher than visual descriptions, likely due to the richer context provided by the manual text.
  • Multi-modal Input is Crucial: Models performed best when provided with both the icon image and its associated context text. Performance dropped notably when either input was omitted, especially for visual descriptions when images were removed.
  • Human Evaluation Confirms Model Rankings: Human evaluations corroborated the automated metrics, ranking Claude 3.5 and GPT-40 as significantly better than LLaVA.

Limitations and Future Work

  • Dataset Scope: The current dataset is limited to icons from HTML manuals. Future work could expand this to include manuals in PDF format, which are more challenging to parse.
  • LLaVA Fine-tuning: The authors plan to fine-tune LLaVA on their dataset to improve its vision encoder and reduce hallucinations.
  • New Metrics: Developing metrics that are more sensitive to hallucinations and better capture visual-likeness aware named entity similarity is a future direction.
  • Generalizability: While the findings are relevant to current LVLMs, the rapid evolution of these models means performance gaps may shift. The importance of multi-modal inputs and challenges with abstract symbols are expected to remain relevant.
  • Abstract Symbol Interpretation: The difficulty in interpreting abstract symbols remains a challenge, and future models may need specialized approaches.

Summary updated

BibTeX
@inproceedings{fletcher2024generating,
  author = {James Fletcher and Nicholas Dehnen and Seyed Nima Tayarani Bathaie and Aijun An and Heidar Davoudi and Ron DiCarlantonio and Gary Farmaner},
  title = {{Generating Vehicular Icon Descriptions and Indications Using Large Vision-Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure