ComRAG: Retrieval-Augmented Generation with Dynamic Vector Stores for Real-time Community Question Answering in Industry

Qinyu Chen, Wenbiao Tao, Zhiwei Zhu, Mingxuan Xi, L. Guo, Yuan Wang, Wei Wang, Yunshi Lan2025 · DOI

Abstract

Community Question Answering (CQA) platforms are essential knowledge repositories in industrial settings, yet they present significant challenges for standard Retrieval-Augmented Generation (RAG) systems. These challenges include the high velocity of incoming data, the high variance in answer quality, and the risk of memory overflow in real-time environments. This paper introduces ComRAG, a novel framework designed to address these issues through a dual-store dynamic memory architecture and a centroid-based clustering mechanism. By distinguishing between high-quality and low-quality historical data and employing a three-path query strategy, ComRAG achieves superior accuracy and efficiency compared to traditional RAG implementations.

Introduction

In industrial environments, Community Question Answering (CQA) platforms—such as internal corporate help desks or public forums like Stack Overflow—serve as dynamic knowledge bases. Unlike static documentation, CQA data is a continuous stream of information where the quality of answers can vary wildly. Traditional RAG systems often struggle in this context because they typically rely on static document sets and lack mechanisms to filter out incorrect or redundant historical answers. Furthermore, as the volume of Q&A pairs grows, retrieval latency and storage costs can become prohibitive.

ComRAG (Community RAG) is designed to bridge the gap between authoritative, static domain knowledge and the evolving, often inconsistent history of community interactions. It introduces a dynamic memory management system that optimizes both the quality of generated responses and the computational efficiency of the retrieval process.

The ComRAG Framework

Dual Dynamic Vector Stores

To manage the variance in community-generated content, ComRAG maintains two distinct databases: a High-Quality Store and a Low-Quality Store. The High-Quality Store contains proven, helpful answers, while the Low-Quality Store archives incorrect or poorly rated responses. This separation allows the system to treat historical data with appropriate levels of confidence.

Centroid-Based Memory Mechanism

To prevent "chunk bloat"—the uncontrolled expansion of the vector database—ComRAG employs a centroid-based clustering approach. Similar questions are grouped around a representative vector, or "centroid." When a new Q&A pair is processed, the system evaluates its similarity to existing centroids. If a new answer is found to be superior to an existing one within the same cluster, it replaces the old entry. This ensures the memory remains compact and populated only by the highest-quality information.

Three-Path Query Strategy

ComRAG optimizes response generation by routing queries through one of three paths based on the similarity of the user's question to historical data:

  1. Direct Reuse: If a new question is nearly identical to a high-quality historical entry, the system serves the existing answer immediately. This bypasses the LLM generation phase, significantly reducing latency.
  2. Referential Generation: If a question is similar but not identical to an HQ entry, the system uses that entry as a "few-shot" example to guide the LLM in generating a tailored response.
  3. Contrastive Generation: If no high-quality match is found, the system retrieves information from static documentation and the Low-Quality Store. It then provides the LLM with a contrastive prompt: "Here is the official documentation, and here is a known incorrect answer to avoid." This use of negative constraints helps the model steer clear of common pitfalls and hallucinations.

Adaptive Temperature Tuning

ComRAG dynamically adjusts the LLM's "temperature" based on the consistency of retrieved snippets. If the retrieved data is consistently high-quality, the temperature is raised to allow for more natural, creative phrasing. Conversely, if the quality is inconsistent, the temperature is lowered to ensure a more conservative and reliable output.

Experimental Evaluation

Datasets

The framework was evaluated on three industrial-scale datasets:

  • MSQA (Microsoft QA): Technical questions from Microsoft forums.
  • ProCQA: Programming-related Q&A pairs from Stack Overflow (Lisp subset).
  • PolarDBQA: A new benchmark based on Alibaba Cloud’s PolarDB documentation, featuring LLM-simulated user inquiries to mimic specialized database domain queries.

Key Results

ComRAG demonstrated significant improvements over baselines such as BM25 and Vanilla RAG:

  • Accuracy: The system achieved up to a 25.9% improvement in semantic similarity (SIM) metrics.
  • Latency: Average query latency was reduced by 8.7% to 23.3%. Notably, on the ProCQA dataset, latency dropped from over 10 seconds to under 5 seconds as the system matured over multiple iterations.
  • Storage Efficiency: The centroid mechanism effectively managed database growth. In ProCQA tests, the chunk growth rate plummeted from 20.23% in the first iteration to just 2.06% by the tenth, proving the system's long-term scalability.

Discussion and Future Work

ComRAG proves that knowing what constitutes a "bad" answer is as valuable as knowing the "good" one. By utilizing the Low-Quality store as a set of negative constraints, the system effectively guides the LLM away from common community errors. Furthermore, the "warm-up" effect—where the system becomes faster as the High-Quality store populates—makes it ideal for long-term industrial deployment.

However, limitations remain. Currently, similarity thresholds are fixed, and the system does not account for the popularity of specific questions. Future research will focus on developing learning-based routers to replace rule-based path selection and implementing dynamic thresholding to further refine memory management.

Summary updated

BibTeX
@misc{chen2025comrag,
  author = {Qinyu Chen and Wenbiao Tao and Zhiwei Zhu and Mingxuan Xi and L. Guo and Yuan Wang and Wei Wang and Yunshi Lan},
  title = {{ComRAG: Retrieval-Augmented Generation with Dynamic Vector Stores for Real-time Community Question Answering in Industry}},
  year = {2025},
  doi = {10.18653/v1/2025.acl-industry.53},
}
Figure