Adopting RAG for LLM-Aided Future Vehicle Design

Vahid Zolfaghari, Nenad Petrovic, Fengjunjie Pan, Krzysztof Lebioda, Alois Knoll

Introduction

The automotive industry is currently navigating a complex "privacy vs. performance" trade-off. While Large Language Models (LLMs) like GPT-4 offer transformative potential for automating software development and system design, their reliance on cloud-based processing presents a significant barrier. For automotive manufacturers, sending sensitive data—including trade secrets, proprietary technical specifications, and customer information—to external cloud providers is often prohibited by strict non-disclosure policies and safety regulations.

While local deployment of massive models is a theoretical solution, the hardware requirements for state-of-the-art proprietary models are prohibitively expensive. This paper investigates an alternative: leveraging Retrieval-Augmented Generation (RAG) to enable smaller, open-source models to perform complex automotive tasks locally and securely, ensuring that proprietary design guidelines and safety standards like ISO 26262 remain within organizational boundaries.

The RAG-Based Framework

The proposed framework utilizes RAG to provide an LLM with a dynamic "look up" capability into a private document database. This allows the model to generate responses based on specific, retrieved context rather than relying solely on its pre-trained weights.

Detailed architecture of the two-stage RAG pipeline including Bi-Encoder retrieval and Cross-Encoder re-ranking.

To ensure scalability and accuracy, the researchers implemented a two-stage retrieval pipeline:

  1. Initial Retrieval: A fast Bi-Encoder model (multi-qa-MiniLM-L6-cos-v1) performs a semantic search across thousands of document chunks, filtering the database down to the top 32 candidates.
  2. Re-Ranking: A more computationally intensive Cross-Encoder model (ms-marco-MiniLM-L-6-v2) evaluates these candidates to select the top 3 most relevant passages, which are then provided as context to the LLM.

Case Studies: Compliance and Design

The effectiveness of this RAG pipeline was evaluated through two primary automotive use cases. The first is a Compliance Chatbot designed to answer technical questions based on a knowledge base of ISO standards (e.g., ISO/TR 4804:2020, ISO 26262-1:2018) and AVC Consortium specifications.

The second use case is a Design Copilot that assists in the development of vehicular systems. This tool integrates RAG with formal verification methods, checking user-defined models (Ecore/OCL) against design guidelines.

Workflow for LLM-based design feedback leveraging RAG and formal constraints check.

As shown in the workflow, the system identifies design failures through constraint checks and then uses the RAG-enhanced LLM to suggest specific corrections based on retrieved standards. This hybrid approach creates a closed-loop system for automated design feedback.

Evaluation and Key Results

The study compared the performance of four models: GPT-4o, Llama 3 (8B), Mistral (7B), and Mixtral (8x7B). Evaluation was conducted using a benchmark of five complex prompts derived from the CeCaS (Central Car Server) project workflow.

  • Reasoning and Accuracy: GPT-4o remains the benchmark for performance, scoring 4.5/5. In contrast, local models like Llama 3 and Mistral scored 2/5, indicating that while they are capable of basic tasks, they still struggle with the most intricate automotive reasoning.
  • Execution Speed: Mistral emerged as the most efficient local option, with an average execution time of 10.93 seconds per prompt. Mixtral was significantly slower at 132.38 seconds.
  • Search Scalability: A critical finding was that while indexing time grows linearly with document volume, search time remains nearly constant at approximately 0.5 seconds, even when the database expands from 900 to over 7,000 pages.

Conclusion

While a "reasoning gap" persists between proprietary cloud models and open-source local models, RAG serves as a vital bridge for the automotive industry. By providing local models with precise context from technical standards, companies can achieve functional utility without compromising data privacy. Future work will focus on fine-tuning these local models on automotive-specific corpora and implementing automated "LLM-as-a-judge" evaluation frameworks to further refine accuracy.

Summary updated

BibTeX
@inproceedings{zolfaghari2024adopting,
  author = {Vahid Zolfaghari and Nenad Petrovic and Fengjunjie Pan and Krzysztof Lebioda and Alois Knoll},
  title = {{Adopting RAG for LLM-Aided Future Vehicle Design}},
  booktitle = {FLLM},
  year = {2024},
  doi = {10.1109/fllm63129.2024.10852467},
}
Figure