Context-Driven Index Trimming: A Data Quality Perspective to Enhancing Precision of RALMs

Kexin Ma, Ruochun Jin, Haotian Wang, Xi Wang, Huan Chen, Yuhua Tang, Qian Wang

Abstract

Retrieval-Augmented Large Language Models (RALMs) have made significant strides in enhancing the accuracy of generated responses. However, existing research often overlooks the data quality issues within retrieval results, often caused by inaccurate existing vector-distance-based retrieval methods. We propose to boost the precision of RALMs’ answers from a data quality perspective through the Context-Driven Index Trimming (CDIT) framework, where Context Matching Dependencies (CMDs) are employed as logical data quality rules to capture and regulate the consistency between retrieved contexts. Based on the semantic comprehension capabilities of Large Language Models (LLMs), CDIT can effectively identify and discard retrieval results that are inconsistent with the query context and further modify indexes in the database, thereby improving answer quality. Experiments demonstrate an average improvement of 3.75% in accuracy on challenging open-domain question-answering tasks. Also, the flexibility of CDIT is verified through its compatibility with various language models and indexing methods, which offers a promising approach to bolster RALMs’ data quality and retrieval precision jointly.

Introduction: The Semantic Proximity Trap

Retrieval-Augmented Generation (RAG) has become the standard for reducing hallucinations in Large Language Models (LLMs). However, it suffers from a fundamental "garbage in, garbage out" problem. Vector databases rely on distance-based metrics (like cosine similarity) to find relevant information.

Conceptual overview of the CDIT framework identifying and pruning inconsistent contexts to improve answer precision.

A critical flaw in this approach is that sentences can be semantically "close" in vector space but factually opposite. For example, "He turned on the radio" and "He turned off the radio" share almost all the same tokens and context, leading retrievers to treat them as highly similar. When a RAG system retrieves such conflicting information, the generator model becomes confused, leading to incorrect answers. CDIT addresses this by focusing on the data quality of the knowledge base itself rather than just the retrieval algorithm.

Methodology: The CDIT Framework

The Context-Driven Index Trimming (CDIT) framework introduces a logical layer to the retrieval process through two main innovations: Context Matching Dependencies (CMDs) and a permanent index trimming mechanism.

Context Matching Dependencies (CMDs)

Inspired by traditional database "Functional Dependencies," CMDs are logical rules that define what constitutes a consistent match. For instance, a rule might state that if two sentences are semantically identical, their subjects, predicates, and objects must align. These rules allow the system to judge whether a retrieved context actually supports the query's intent.

The Trimming Mechanism

The workflow of CDIT involves a feedback loop that cleans the database index over time:

The six-step workflow of CDIT, illustrating the path from initial retrieval to permanent index updates.
  1. Extraction: An advanced LLM (like GPT-3.5-turbo) acts as a "judge" to decompose retrieved sentences into linguistic components (subject, predicate, object).
  2. Judgment: The judge checks these components against the query using the CMD rules. If a retrieved context contradicts the query (e.g., different actions on the same object), it is flagged as inconsistent.
  3. The Witness Theorem: This is the core logic of the framework. If a query $q$ is consistent with sentence $s_1$ but inconsistent with $s_2$, the query acts as a "witness" that $s_1$ and $s_2$ should not be linked in the vector index.
  4. Index Pruning: Once enough "witnesses" are collected, CDIT permanently modifies the vector database index (like an HNSW graph) by cutting the edges between these inconsistent vectors. This ensures that future queries won't retrieve the same conflicting data.

Evaluation and Results

The framework was evaluated on four challenging factual question-answering benchmarks: ARC-Challenge, PubHealth, PopQA, and TriviaQA-unfiltered, using an English Wikipedia dump as the knowledge base.

  • Broad Accuracy Gains: CDIT achieved an average accuracy improvement of 3.75% across seven different base models (including Llama-7b, Mistral-7b, and Falcon-7b).
  • Peak Performance: The most significant single-task improvement reached 15.21% (Falcon-7b on TriviaQA).
  • Index Compatibility: The method proved effective across various indexing strategies, including IndexFlatL2, IndexHNSWFlat, and IndexIVFFlat.
  • Synergy with Self-RAG: When integrated with Self-RAG, CDIT provided an additional 3.62% average boost, proving it complements existing retrieval-improvement methods.

Key Insights

  • Semantic Proximity ≠ Semantic Consistency: High vector similarity does not guarantee factual relevance. CDIT proves that "fuzzy" vector search needs a logical layer to handle nuances like negation or conflicting actions.
  • Index Trimming as Permanent Cleaning: Unlike "re-ranking" methods that filter results at runtime, CDIT actually cleans the database. By pruning the index, the system becomes more efficient and accurate over time as it "learns" which data points are misleadingly close to each other.
  • Solving the "Information Explosion": As the number of retrieved documents ($top-k$) increases, standard RAG performance often plateaus or drops because of noise. CDIT effectively mitigates this, showing significant gains even when $top-k$ is as high as 50.

Limitations and Future Work

While powerful, CDIT has limitations. Decomposing long, complex paragraphs into simple triples is difficult and can lead to errors. Currently, CMD rules are hand-crafted; future work should focus on automatically discovering these rules from data. Additionally, the reliance on high-performing LLMs for judgment introduces latency, which could be mitigated by exploring local lexical analyzers or dependency parsers.

Conclusion

CDIT offers a promising approach to bolster RALMs’ data quality and retrieval precision jointly. By treating retrieval as a data quality problem and applying logical pruning to the index, it provides a robust mechanism for ensuring that LLMs are grounded in consistent, high-quality information.

Summary updated

Abstract

Retrieval-Augmented Large Language Models(RALMs) have made significant strides in enhancing the accuracy of generated responses. However, existing research often overlooks the data quality issues within retrieval results, often caused by inaccurate existing vector-distance-based retrieval methods. We propose to boost the precision of RALMs’ answers from a data quality perspective through the Context-Driven Index Trimming (CDIT) framework, where Context Matching Dependencies (CMDs) are employed as logical data quality rules to capture and regulate the consistency between retrieved contexts. Based on the semantic comprehension capabilities of Large Language Models (LLMs), CDIT can effectively identify and discard retrieval results that are inconsistent with the query context and further modify indexes in the database, thereby improving answer quality. Experiments demonstrate average improvement of 3.75% in accuracy on challenging open-domain question-answering tasks. Also, the flexibility of CDIT is verified through its compatibility with various language models and indexing methods, which offers a promising approach to bolster RALMs’ data quality and retrieval precision jointly.
BibTeX
@inproceedings{ma2024context,
  author = {Kexin Ma and Ruochun Jin and Haotian Wang and Xi Wang and Huan Chen and Yuhua Tang and Qian Wang},
  title = {{Context-Driven Index Trimming: A Data Quality Perspective to Enhancing Precision of RALMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=0BJAsF2hUX},
}
Figure