TL;DR
This paper tackles the challenge of providing intuitive, landmark-based answers to questions about car component locations. They achieve this by generating Spatial Knowledge Graphs (SKGs) from vehicle dashboard diagrams using Large Vision-Language Models (LVLMs). Their novel "Per-Component" pipeline and "Significance" scoring method significantly improve landmark identification, leading to more helpful answers for in-car virtual assistants.
What Problem Does This Paper Address?
Imagine asking your car's virtual assistant, "Where is the Vehicle Stability Assist button?" A naive answer might be, "It's to the right of the Lane Keeping Assist button." This is unhelpful if you don't know where the Lane Keeping Assist button is. The core problem is that current systems struggle to provide answers grounded in easily recognizable, common reference points (landmarks). This stems from two main issues: 1) identifying which components are salient enough to be good landmarks, and 2) accurately discerning precise spatial relationships from complex, often monochrome technical diagrams like those found in car manuals. Existing methods often fail due to the unique characteristics of these diagrams (lack of color, texture, and reliance on specific annotation styles).
Methods
The authors propose generating a Spatial Knowledge Graph (SKG) from a vehicle's dashboard diagram and its legend. This SKG explicitly maps components and their spatial relationships. To build this graph, they explore three LVLM-based generation pipelines:
- Per-Attribute Pipeline: Decomposes the task by querying the LVLM for each attribute (Size, Frequency, Proximity, Position) separately for all components. This generates small outputs, suitable for models with limited token windows.
- Per-Component Pipeline: Iteratively focuses on one "source" component at a time, prompting the LVLM for a comprehensive analysis of its attributes and relationships to all other components. This balances decomposition with contextual richness.
- Single-Prompt Pipeline: Attempts to generate the entire SKG in a single query, testing the limits of models with large context windows.
A key innovation is their algorithmic Significance score. This score, a weighted average of component attributes (Size, Proximity, Position, and optionally Frequency), programmatically identifies the most salient components to serve as landmarks. This score is rooted in cognitive heuristics for human navigation, prioritizing visually and cognitively prominent features.
Datasets
The authors created a new, comprehensive evaluation dataset containing 15,852 human-annotated spatial relationships from the dashboards of 65 vehicles across 9 manufacturers. This dataset was specifically designed to test robustness across diverse diagrammatic styles. The ground truth for this dataset was produced by a professional annotator.
Key Insights and Ideas
- Decomposition is Key: The "useless reference" problem highlights the need for structured spatial understanding, which LVLMs alone struggle to provide directly from images.
- Per-Component Wins: The decomposed "Per-Component" pipeline, when paired with Gemini 2.5 Flash, proved most effective for generating high-quality SKGs.
- Visual Cues Trump Behavior: Surprisingly, core visual and geometric properties (Size, Proximity, Position) were better predictors of landmark utility than the estimated behavioral 'Frequency' attribute. This suggests that for landmark identification in technical diagrams, what you see is more important than how often you'd use it.
- Stylistic Variance Matters: The performance of the generation pipelines is significantly influenced by the stylistic characteristics of the diagrams (e.g., clean line drawings vs. photo-realistic renderings, complexity of leader lines). This is a crucial factor for generalization.
Key Results
- The best-performing SKG generation method (Per-Component pipeline with Gemini 2.5 Flash, excluding Frequency) identified landmarks with 71.3% agreement with human annotators, as measured by their Significance score.
- In downstream question answering, the KG-augmented approach significantly outperformed an image-only baseline across multiple metrics (LLM judge, ROUGE-1, BERTScore), demonstrating the value of structured spatial knowledge. The LLM Judge showed the largest performance gap, indicating its effectiveness in evaluating semantic alignment.
- Models performed better on clean, high-contrast line drawings and worse on photo-realistic, shaded renderings or those with complex, overlapping leader lines.
Limitations and Future Work
- Single Annotator: The ground truth was produced by a single expert annotator, which could introduce bias. Future work should involve multiple annotators and inter-annotator agreement studies.
- Labeled Legends: The method assumes the availability of a labeled legend. Extending it to unlabeled diagrams or real-world photos is a significant challenge.
- Curated Questions: The QA evaluation used a curated set of 10 questions per diagram, which may not capture the full range of user queries.
- Broader Applications: The generated SKGs could support a wider range of diagram reasoning tasks beyond location-based QA, such as technician training or augmented reality systems. Future work could explore domain adaptation and model fine-tuning for specific diagram styles.
Summary updated