Querying Large Automotive Software Models: Agentic vs. Direct LLM Approaches

Łukasz Mazur, Nenad Petrović, James Pontes Miranda, Ansgar Radermacher, Robert Rasche, Alois Knoll

Abstract

As automotive software systems grow in complexity, engineers increasingly rely on large-scale models like AUTOSAR or Ecore for system design. While Large Language Models (LLMs) provide a promising natural language interface for querying these models, the sheer size of industrial software models often exceeds context window limits or incurs prohibitive token costs. This paper evaluates two paradigms: direct full-context prompting and an agentic, tool-augmented workflow. Our findings demonstrate that while top-tier models maintain high accuracy in both setups, the agentic approach reduces token consumption by over 99%, facilitating the deployment of local, privacy-preserving LLMs in sensitive engineering environments.

Introduction

In the automotive industry, software engineering is governed by complex, highly structured models. Navigating these models—often containing thousands of classes and lines of XML-based code—is a significant bottleneck for developers. Large Language Models (LLMs) offer a potential solution by allowing engineers to query these models using natural language.

However, two major hurdles remain. First, "direct prompting"—where the entire model is fed into the LLM's context—is often impossible or extremely expensive due to token limits. Second, the automotive sector's strict privacy and intellectual property (IP) requirements make cloud-based LLM APIs unattractive. There is a strong preference for smaller, "local" models that can run on-premise. This study investigates whether an agentic approach, where an LLM uses a suite of tools to browse a model incrementally, can match the performance of full-context prompting at a fraction of the cost.

Methodology: Direct vs. Agentic Architectures

The researchers compared two primary architectures for model querying:

  1. Direct Full-Context Prompting: This baseline approach involves loading the entire software model (in Ecore format) directly into the LLM's prompt. The model is then asked to answer a specific question zero-shot.
  2. Agentic ReAct Architecture: In this setup, the LLM is not given the file content initially. Instead, it is provided with a directory path and a set of IDE-like tools. The agent must iteratively plan, execute tool calls, and observe the results to find the necessary information.
Comparison of Direct Full-Context Prompting (a) and Agentic Tool-Augmented Workflows (b).

The agentic workflow is limited to 100 iterations. To simulate realistic local deployment scenarios, the authors tested several "mini" models, including GPT-4o mini, GPT-4.1 mini, o4-mini, and Gemini 2.5 Flash Preview.

The Agentic Toolkit

To enable the agentic workflow, the LLM was provided with a specific set of file-manipulation tools. These tools allow the model to explore the file structure and read specific segments of the code without loading the entire document into memory.

Description of the file-access tools available to the agentic LLM.

By using tools like search_file and open_file (which reads 50-line windows), the agent can pinpoint relevant sections of the 13,500-line Ecore model.

Experimental Setup and Datasets

The evaluation utilized an open-source INCHRON am2inc Ecore metamodel, a standard for timing analysis in multi-core embedded systems. The model spans approximately 13,500 lines, translating to roughly 118,000 tokens.

The authors developed a benchmark of 20 queries categorized by complexity:

  • Direct Class Inspection: Simple lookups of class properties.
  • Single/Multiple Inheritance Chains: Queries requiring the agent to trace relationships between classes.
  • Semantic Queries: Questions requiring an understanding of the model's underlying logic.

Key Results and Insights

The study revealed a dramatic disparity in efficiency and varying levels of robustness across models.

Token Efficiency

The most striking result was the reduction in token usage. Direct prompting required between 118,000 and 137,000 tokens per query. In contrast, the agentic approach consumed only 640 to 780 tokens—a reduction of over 99%. This efficiency makes the agentic approach highly compatible with local LLM deployments where memory and compute are limited.

Accuracy and Model Capability

While efficiency was high, accuracy depended heavily on the model's reasoning capabilities:

  • High-Performance Models: GPT-4.1 mini and o4-mini maintained a consistent 90% accuracy in both direct and agentic modes.
  • Performance Degradation: Weaker models struggled significantly with tool use. Gemini 2.5 Flash dropped from 80% (direct) to 40% (agentic), while GPT-4o mini plummeted from 45% to just 10%.

Failure Analysis

Failures in the agentic setup were typically caused by "reasoning loops" or incomplete exploration. For example, an agent might find a parent class but fail to recursively check if that parent inherited properties from another class. Additionally, some models relied too heavily on generic scroll_down commands rather than efficient search_file calls, eventually hitting the 100-iteration limit.

Conclusion

The agentic approach proves to be a viable and highly efficient method for querying massive software models, provided the underlying LLM has strong reasoning and tool-use capabilities. By reducing token overhead by two orders of magnitude, this workflow enables the use of smaller, local models for complex engineering tasks, addressing both cost and privacy concerns in the automotive industry. Future work will focus on expanding the toolkit to include domain-specific functions and enabling the agent to modify models while maintaining architectural consistency.

Summary updated

Abstract

Large language models (LLMs) offer new opportunities for interacting with complex software artifacts, such as software models, through natural language. They present especially promising benefits for large software models that are difficult to grasp in their entirety, making traditional interaction and analysis approaches challenging. This paper investigates two approaches for leveraging LLMs to answer questions over software models: direct prompting, where the whole software model is provided in the context, and an agentic approach combining LLM-based agents with general-purpose file access tools. We evaluate these approaches using an Ecore metamodel designed for timing analysis and software optimization in automotive and embedded domains. Our findings show that while the agentic approach achieves accuracy comparable to direct prompting, it is significantly more efficient in terms of token usage. This efficiency makes the agentic approach particularly suitable for the automotive industry, where the large size of software models makes direct prompting infeasible, establishing LLM agents as not just a practical alternative but the only viable solution. Notably, the evaluation was conducted using small LLMs, which are more feasible to be executed locally-an essential advantage for meeting strict requirements around privacy, intellectual property protection, and regulatory compliance. Future work will investigate software models in diverse formats, explore more complex agent architectures, and extend agentic workflows to support not only querying but also modification of software models.
BibTeX
@misc{mazur2025querying,
  author = {Łukasz Mazur and Nenad Petrović and James Pontes Miranda and Ansgar Radermacher and Robert Rasche and Alois Knoll},
  title = {{Querying Large Automotive Software Models: Agentic vs. Direct LLM Approaches}},
  year = {2025},
  eprint = {2506.13171},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.13171},
}
Figure