Vortex under Ripplet: An Empirical Study of RAG-enabled Applications

Yuchen Shao, Yuheng Huang, Jiawei Shen, Lei Ma, Ting Su, Chengcheng Wan

Abstract

Large Language Models (LLMs) are increasingly integrated into complex software systems via Retrieval-Augmented Generation (RAG). However, these integrations often introduce unique defects that compromise system reliability. This paper presents a comprehensive empirical study of 100 open-source RAG-enabled applications, identifying 18 distinct defect patterns across four architectural layers. Our analysis of over 3,000 issue reports reveals that 77% of these applications suffer from multiple integration-related bugs, highlighting a significant gap between model performance and system-level robustness.

Introduction

While LLMs demonstrate remarkable capabilities in isolation, integrating them into traditional software stacks is notoriously difficult. Developers often treat LLMs as "black boxes," leading to a disconnect between a model that performs well in a playground environment and a system that is reliable in production.

The authors identify three core integration challenges:

  1. Lacking Interface Specifications: Unlike traditional APIs with strict types, LLMs produce unstructured text, making it difficult to define boundaries or validate "correct" behavior.
  2. Software Context Sensitivity: General-purpose LLMs often fail to adhere to specific domain requirements without meticulous prompt and RAG orchestration.
  3. Complex System Management: RAG workflows are resource-intensive (GPU memory, token costs) and introduce new security vulnerabilities such as prompt injection.

Methodology

The researchers conducted a multi-step empirical study focusing on 100 non-trivial, open-source LLM-enabled applications from GitHub. These projects spanned various domains, including QA systems, task management tools, and chatbots. The majority of these applications (82%) utilized GPT-based models, while ChromaDB (42%) and Pinecone (21%) were the most common vector databases.

Overview of the RAG/VectorDB system architecture and interaction flow.

The team manually reviewed over 3,000 issue reports to identify "integration failures"—defined as bugs fixable by changing how components interact rather than modifying the underlying AI algorithms. Through an iterative process, they categorized these failures into a taxonomy of 18 defect patterns across four architectural layers: the LLM Agent, the Vector Database, general Software Components, and the System layer. These findings were compiled into HYDRANGEA, an open-source defect library containing 546 identified bugs.

Taxonomy of RAG Integration Defects

The study reveals that most "AI failures" are actually software engineering failures. Key patterns identified include:

  • The Context Limit Trap: 85% of studied apps fail to manage the LLM's token limit properly, often sending massive document chunks that result in truncated or nonsensical responses.
  • Semantic Overwriting: In vector databases, developers often mistakenly use the same semantic vector (e.g., just a title) for different content, leading to data loss where new knowledge entries overwrite old ones.
  • The "Infinite Loop" of Agents: In multi-turn applications, the lack of an "on-exit" condition can lead to infinite refinement loops that never provide a final answer.
  • GPU Memory Leaks: Local LLM apps frequently fail to explicitly release GPU memory when switching models, leading to out-of-memory (OOM) crashes.

Key Results

The empirical analysis underscores the ubiquity of these defects. A staggering 77% of applications contained more than three types of integration defects.

Example of expected structured output versus common verbosity issues in LLM responses.

Input/output mismatches were particularly prevalent: 83% of apps lacked proper input format validation, and 26% struggled with incompatible output formats, such as the LLM failing to return valid JSON. Retrieval precision also remains a significant hurdle, with 45% of apps suffering from imprecise knowledge retrieval due to vague queries. Furthermore, 53% of applications that invoked system calls were found to have security gaps, including vulnerabilities to text-to-code injection attacks.

Conclusion and Future Work

The study demonstrates that the bottleneck for RAG-enabled applications is often the integration layer rather than the model itself. The authors propose moving toward automated detection and repair of these 18 patterns. By implementing static analysis and "self-correcting" LLM prompts, developers can resolve many of these integration hurdles automatically, leading to more robust and secure AI-enabled systems.

Summary updated

BibTeX
@inproceedings{shao2025vortex,
  author = {Yuchen Shao and Yuheng Huang and Jiawei Shen and Lei Ma and Ting Su and Chengcheng Wan},
  title = {{Vortex under Ripplet: An Empirical Study of RAG-enabled Applications}},
  booktitle = {ICSE},
  year = {2025},
  doi = {10.1109/icse55347.2025.00204},
}
Figure