Abstract
This paper presents a multi-dimensional optimization of Retrieval-Augmented Generation (RAG) specifically tailored for the automotive industry. Utilizing locally deployed Ollama models, the study addresses the challenges of processing complex technical documentation while maintaining strict data privacy. By combining advanced PDF parsing, hybrid retrieval, and a self-reflective agentic workflow, the proposed system significantly improves the accuracy and reliability of chatbots. Evaluation using the RAGAS framework shows a 9.4% improvement in answer relevancy and a 5.5% increase in faithfulness over baseline models.
Introduction
The automotive industry relies on a massive volume of complex technical documents characterized by multi-column layouts, intricate tables, and specialized terminology. Standard "out-of-the-box" RAG systems often struggle with these documents, leading to lost logical flow and inaccurate information retrieval. Furthermore, the sensitive nature of proprietary designs necessitates local deployment to ensure data privacy, often on hardware with limited computational resources. This research explores methods to optimize RAG performance under these specific industrial constraints.
Methods
The researchers developed a four-pronged optimization strategy built on the LangChain and LangGraph frameworks, focusing on local execution via Ollama.
Enhanced PDF Processing
To overcome the limitations of standard parsers, a custom algorithm was implemented using PDFMiner for layout extraction and Tabula for structured table data. This process converts complex PDFs into clean Markdown, preserving the structural relationships between text and tables that are often lost in plain text extraction.
Advanced RAG Pipeline and Agentic Workflow
The system utilizes a hybrid retrieval approach combining BM25 keyword-based search with vector embeddings, followed by a BGE Reranker model to prioritize contextually accurate snippets.
Beyond the standard retrieval chain shown above, the authors implemented a "Self-RAG" agentic workflow using LangGraph. This Directed Acyclic Graph (DAG) allows the system to:
- Grade retrieved documents for relevance.
- Rewrite queries if initial retrieval fails.
- Reflect on and self-correct generated answers.
To optimize performance on local hardware, a custom ChatFunction class was developed to utilize Ollama’s JSON mode, allowing the model to dynamically adjust response detail based on query complexity.
Datasets
The system was evaluated across three distinct datasets:
- QReCC: 10,000 multi-turn conversational questions.
- CoQA: Over 8,000 conversational dialogues.
- Automotive Industry Dataset: A proprietary benchmark consisting of technical reports, manufacturing guidelines, and quality control procedures from a leading manufacturer.
Key Results
Performance was measured using the RAGAS framework, focusing on Faithfulness, Answer Relevancy, Context Precision, and Context Recall.
- Relevancy Gains: The optimized pipeline showed a 7.6% relative improvement in answer relevancy on the automotive dataset compared to a naive RAG baseline.
- Agentic Superiority: The Self-RAG agent with custom function calling outperformed the baseline by 9.4% in answer relevancy and 5.5% in faithfulness.
- Robustness: Improvements were consistent across both specialized automotive data and general public datasets (QReCC and CoQA).
Discussion and Conclusion
The study highlights that converting PDFs to Markdown is a critical step for RAG in technical domains, as it allows LLMs to interpret the hierarchy and structure of the data. Furthermore, the results demonstrate that smaller, locally deployed models can outperform larger cloud-based models when augmented with self-reflective agentic loops and specialized rerankers.
Future work will focus on reducing the latency of the agentic loops for real-time industrial applications and expanding the system's multi-modal capabilities to interpret technical diagrams and schematics directly.
Summary updated