How Can Personalized Context Help? Exploring Joint Retrieval of Passage and Personalized Context

Hui Wan, Hongkang Li, Songtao Lu, Xiaodong Cui, Marina DanilevskyICASSP 2024 · DOI

Introduction

While Large Language Models (LLMs) have revolutionized information retrieval, they often struggle with queries that are inherently underspecified. In real-world scenarios, a user's question—such as "Am I eligible for this rebate?"—cannot be answered by a general knowledge base alone. It requires two distinct pieces of information: the external document containing the rebate rules and the specific personalized context of the user (e.g., their location, employment status, or history).

Standard Retrieval-Augmented Generation (RAG) systems typically treat queries in a vacuum, ignoring the nuanced interplay between documents and user-specific data. This paper introduces the task of context-aware passage retrieval and proposes Personalized Context-Aware Search (PCAS), a framework designed to jointly identify the most relevant document and the specific piece of personalized context needed to provide an accurate answer.

The Challenge of Personalized Retrieval

Most existing RAG systems attempt to handle personalization by either ignoring it or by prepending all available user data to the prompt. However, simply "cramming" all context into a prompt creates significant noise and often exceeds token limits, leading to degraded performance.

The core challenge lies in the fact that the relevance of a document is often conditional on the context, and vice versa. A query might be vague, but a relevant document contains specific "slots" (like eligibility criteria) that only the correct piece of context can fill. Therefore, a system must effectively balance document relevance with contextual fit to disambiguate the user's intent.

Proposed Method: PCAS

To address this, the authors propose Personalized Context-Aware Search (PCAS). Unlike sequential baselines that retrieve a document and then look for context (or the reverse), PCAS treats the document and context as a pair to be optimized jointly. The process follows three main steps:

  1. Candidate Generation: The system uses the user query ($q$) to retrieve the top-$K$ document candidates ($d$) from a large corpus using standard dense retrieval.
  2. Context Selection: For each candidate document, the system identifies the most relevant piece of context ($c$) from the user's available context set (e.g., a set of personal facts or history snippets).
  3. Joint Scoring: PCAS calculates a final ranking score using a convex combination: $$\text{Score} = \lambda \cdot \text{score}(d, q) + (1 - \lambda) \cdot \text{score}(d, c)$$

By adjusting the hyperparameter $\lambda$, the system can prioritize query relevance or contextual alignment. This approach allows the document itself to act as a bridge, helping the system determine which piece of context is actually necessary to answer the query.

The ORCA-ShARC Benchmark

To evaluate this task, the researchers introduced ORCA-ShARC (Open Retrieval Context-Aware ShARC). This benchmark is adapted from the ShARC and OR-ShARC datasets, which focus on conversational QA based on rule documents.

The authors modified these datasets to create a more challenging retrieval environment. While the original data provided a single "gold" context, ORCA-ShARC expands this into a "Scenario Set" containing 10 pieces of context per example. Only one is relevant; the other nine are distractors sampled from the dataset that do not explicitly contradict the gold context but are irrelevant to the specific query-document pair. The final dataset includes 651 rule snippets and nearly 18,000 training examples.

Key Insights and Results

The researchers evaluated PCAS against three primary baselines: B1 (concatenating all context to the query), B2 (retrieving the document first, then the context), and B3 (retrieving the context first, then the document). Testing across various neural architectures—including ColBERT, DPR, ANCE, and S-BERT—yielded several key findings:

  • PCAS Outperforms Baselines: PCAS consistently achieved higher Recall@1 and Mean Average Precision (MAP). For example, using ColBERT, PCAS achieved a Recall@1 of 59.19%, significantly outperforming the "all-in" B1 baseline (34.93%) and the sequential B2 baseline (55.57%).
  • The Danger of Noise: Baseline B1 performed the worst, confirming that simply increasing the context window is not a silver bullet. Irrelevant context acts as noise that confuses the retrieval model.
  • Model Agnosticism: The performance gains were consistent across different dense retriever architectures, suggesting that the PCAS framework is robust and widely applicable.
  • Effective Context Selection: PCAS proved highly effective at isolating the "gold" context from distractors, approaching the performance of an "Oracle" system that is given the correct context upfront.

Limitations and Future Directions

While PCAS represents a significant step forward for personalized RAG, the authors note several areas for future work:

  1. Generation Evaluation: The study focuses on the retrieval stage; future research should evaluate how improved retrieval translates to the quality of LLM-generated text.
  2. Context Complexity: The current benchmark uses relatively short text snippets. Real-world context may be structured (e.g., JSON logs) or significantly more voluminous.
  3. Integration: Future iterations could explore integrating the PCAS scoring mechanism directly into the training process of dense retrievers or incorporating human feedback to further refine context selection.

Conclusion

PCAS demonstrates that document and context retrieval are deeply intertwined. By treating them as a joint optimization problem, systems can better disambiguate user queries and provide more accurate, personalized responses without being overwhelmed by contextual noise.

Summary updated

BibTeX
@inproceedings{wan2024how,
  author = {Hui Wan and Hongkang Li and Songtao Lu and Xiaodong Cui and Marina Danilevsky},
  title = {{How Can Personalized Context Help? Exploring Joint Retrieval of Passage and Personalized Context}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10447921},
}

Loading related content...

Figure