Abstract
As automotive software systems grow in complexity, the reliance on Model-Based Engineering (MBE) has increased, utilizing visual formalisms like Unified Modeling Language (UML) and Eclipse Modeling Framework (EMF). However, the manual interpretation of these information-dense diagrams remains a significant bottleneck. This paper investigates the potential of Multimodal Large Language Models (MLLMs) to automate the summarization and analysis of engineering diagrams. By benchmarking several state-of-the-art models against a specialized automotive metamodel, the study identifies current capabilities in component identification and highlights persistent challenges in relationship mapping and difference detection. Notably, the open-source InternVL2-8B-MPO model demonstrated superior performance in industrial reasoning tasks compared to larger, closed-source counterparts.
Introduction
In the domain of automotive software development, system designs are traditionally captured through formal visual models. These diagrams serve as the blueprint for complex architectures, containing intricate hierarchies, functional descriptions, and relationship chains. While standard Large Language Models (LLMs) have revolutionized text-based requirement processing, they remain effectively "blind" to the visual logic inherent in MBE diagrams.
This research, conducted by the Technical University of Munich, explores whether MLLMs can bridge this gap. The goal is to provide automated multimodal summarization that can assist engineers in maintaining, updating, and verifying complex system designs without the manual overhead currently required to parse visual documentation.
Methodology
The researchers conducted a comparative evaluation of leading MLLMs, categorized into closed-source models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Gemini 2.0 Flash) and open-source models (Llama 3.2, InternVL2, and Qwen2 VL).
The evaluation was structured around five Research Questions (RQs) designed to test a gradient of cognitive and structural complexity:
- Identification: Can the model list all classes present in a diagram?
- Functional Understanding: Can the model extract specific properties and functions within a defined class?
- Classification: Can the model correctly categorize component types (e.g., identifying an FPGA as a Co-Processor)?
- Relationship Mapping: Can the model trace chains of relationships between distant components?
- Difference Detection: Can the model compare two similar diagrams and identify additions or removals?
The study utilized a Centralized Car Server Metamodel as the primary case study. Ground truths were manually established for five specific prompt-based questions (Q1-Q5) to ensure rigorous scoring.
Key Insights and Results
Performance Across Tasks
The benchmarking revealed a sharp divide between basic visual recognition and complex logical reasoning. For simple tasks like Identification (Q1), most models achieved near-perfect accuracy, successfully identifying nearly 30 distinct classes within a single diagram.
However, performance diverged significantly in Functional Understanding (Q2) and Relationship Mapping (Q4). For instance, while Claude 3.5 and InternVL2 provided accurate functional descriptions, Gemini 2.0 struggled, occasionally providing entirely incorrect responses. Relationship mapping proved particularly difficult; many models suffered from hallucinations, generating plausible-sounding but logically incorrect paths between components.
The "Difference Detection" Hurdle
The most challenging task was Difference Detection (Q5). Most high-end models, including GPT-4o and Claude 3.5, failed to accurately identify changes between two similar UML diagrams. While they could describe each diagram in isolation, they lacked the comparative reasoning to "spot the difference" when specific classes were removed or modified.
The Success of InternVL2-8B-MPO
A significant finding of the study was the performance of InternVL2-8B-MPO. Despite having only 8 billion parameters, it was the only model to answer all five categories correctly, including the difficult difference detection task. Researchers attribute this success to the model's robust "middleware" layer between its visual and linguistic encoders, which appears better optimized for structural and spatial reasoning than the architectures of much larger models.
Industrial Application and Proof of Concept
To demonstrate practical utility, the authors proposed a workflow where an MLLM (InternVL2) extracts updates from diagrams, which are then passed to a text-based LLM (Llama3-8B). This secondary model ensures compliance with ISO standards using Object Constraint Language (OCL).
From a performance standpoint, the system achieved manageable latencies on an L4 GPU, with execution times ranging from ~2.6 seconds for simple sensor extraction to ~9.1 seconds for complex difference detection. This suggests that such systems are viable for near-real-time engineering support.
Conclusion and Future Work
The study concludes that while MLLMs show immense promise for MBE, they are not yet a "drop-in" replacement for human oversight, particularly regarding complex relationship tracing. The success of smaller, open-source models like InternVL2 is highly relevant for the automotive industry, where data privacy concerns often necessitate local, on-premise deployment of AI tools.
Future research will focus on scaling these capabilities to handle large sets of interconnected diagrams and exploring how automated summaries can be integrated directly into hardware abstraction layers and code generation pipelines. There remains a critical need for specialized, industry-grade benchmarks to further refine MLLM performance in engineering contexts.
Summary updated